Jul 07 15:38:28.688723 ip-10-0-139-248 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 07 15:38:28.688738 ip-10-0-139-248 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 07 15:38:28.688748 ip-10-0-139-248 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 07 15:38:28.689108 ip-10-0-139-248 systemd[1]: Failed to start Kubernetes Kubelet. Jul 07 15:38:38.866572 ip-10-0-139-248 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 07 15:38:38.866590 ip-10-0-139-248 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot b4d0f257b3f543abbb11cf3b1144ec48 -- Jul 07 15:40:15.693427 ip-10-0-139-248 systemd[1]: Starting Kubernetes Kubelet... Jul 07 15:40:16.140484 ip-10-0-139-248 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 15:40:16.140484 ip-10-0-139-248 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 15:40:16.140484 ip-10-0-139-248 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 15:40:16.140484 ip-10-0-139-248 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 15:40:16.140484 ip-10-0-139-248 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 15:40:16.141603 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.141487 2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 15:40:16.148378 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148358 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:40:16.148378 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148377 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:40:16.148451 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148383 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:40:16.148451 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148386 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:40:16.148451 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148390 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:40:16.148451 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148393 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:40:16.148451 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148396 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:40:16.148451 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148399 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:40:16.148451 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148404 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:40:16.148451 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148408 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:40:16.148451 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148411 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:40:16.148451 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148413 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:40:16.148451 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148416 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:40:16.148451 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148418 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:40:16.148451 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148427 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:40:16.148451 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148430 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:40:16.148451 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148433 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:40:16.148451 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148435 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:40:16.148451 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148437 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:40:16.148451 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148440 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:40:16.148451 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148443 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:40:16.148891 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148446 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:40:16.148891 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148448 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:40:16.148891 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148451 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:40:16.148891 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148453 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:40:16.148891 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148456 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:40:16.148891 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148459 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:40:16.148891 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148464 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:40:16.148891 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148468 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:40:16.148891 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148471 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:40:16.148891 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148474 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:40:16.148891 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148477 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:40:16.148891 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148480 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:40:16.148891 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148483 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:40:16.148891 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148486 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:40:16.148891 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148490 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:40:16.148891 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148492 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:40:16.148891 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148496 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:40:16.148891 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148500 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:40:16.148891 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148503 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:40:16.149362 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148512 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:40:16.149362 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148514 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:40:16.149362 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148517 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:40:16.149362 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148519 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:40:16.149362 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148522 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:40:16.149362 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148524 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:40:16.149362 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148527 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:40:16.149362 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148529 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:40:16.149362 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148531 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:40:16.149362 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148540 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:40:16.149362 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148543 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:40:16.149362 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148545 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:40:16.149362 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148548 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:40:16.149362 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148551 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:40:16.149362 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148554 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:40:16.149362 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148557 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:40:16.149362 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148560 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:40:16.149362 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148562 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:40:16.149362 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148565 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:40:16.149362 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148567 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:40:16.149857 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148570 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:40:16.149857 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148572 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:40:16.149857 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148575 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:40:16.149857 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148578 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:40:16.149857 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148580 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:40:16.149857 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148584 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:40:16.149857 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148586 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:40:16.149857 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148589 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:40:16.149857 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148591 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:40:16.149857 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148593 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:40:16.149857 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148596 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:40:16.149857 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148598 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:40:16.149857 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148601 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:40:16.149857 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148604 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:40:16.149857 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148606 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:40:16.149857 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148609 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:40:16.149857 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148611 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:40:16.149857 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148614 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:40:16.149857 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148616 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:40:16.149857 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148618 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:40:16.150357 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148621 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:40:16.150357 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148624 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:40:16.150357 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148626 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:40:16.150357 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148629 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:40:16.150357 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148631 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:40:16.150357 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148634 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:40:16.150357 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148638 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:40:16.150357 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148641 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:40:16.150357 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148643 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:40:16.150357 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148646 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:40:16.150357 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148648 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:40:16.150357 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148651 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:40:16.150357 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148654 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:40:16.150357 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148656 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:40:16.150357 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148659 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:40:16.150357 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148662 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:40:16.150357 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148664 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:40:16.150357 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148667 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:40:16.150357 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.148669 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:40:16.152157 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152144 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:40:16.152157 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152156 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:40:16.152220 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152160 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:40:16.152220 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152163 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:40:16.152220 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152167 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:40:16.152220 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152170 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:40:16.152220 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152173 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:40:16.152220 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152176 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:40:16.152220 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152179 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:40:16.152220 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152182 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:40:16.152220 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152185 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:40:16.152220 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152188 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:40:16.152220 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152191 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:40:16.152220 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152194 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:40:16.152220 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152196 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:40:16.152220 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152198 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:40:16.152220 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152201 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:40:16.152220 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152204 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:40:16.152220 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152207 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:40:16.152220 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152209 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:40:16.152220 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152212 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:40:16.152220 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152214 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:40:16.152699 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152217 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:40:16.152699 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152219 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:40:16.152699 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152222 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:40:16.152699 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152225 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:40:16.152699 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152228 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:40:16.152699 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152231 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:40:16.152699 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152234 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:40:16.152699 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152236 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:40:16.152699 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152239 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:40:16.152699 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152241 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:40:16.152699 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152244 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:40:16.152699 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152246 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:40:16.152699 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152249 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:40:16.152699 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152251 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:40:16.152699 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152254 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:40:16.152699 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152257 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:40:16.152699 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152259 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:40:16.152699 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152261 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:40:16.152699 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152270 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:40:16.153183 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152275 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:40:16.153183 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152277 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:40:16.153183 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152280 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:40:16.153183 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152290 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:40:16.153183 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152293 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:40:16.153183 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152297 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:40:16.153183 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152300 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:40:16.153183 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152303 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:40:16.153183 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152305 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:40:16.153183 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152308 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:40:16.153183 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152310 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:40:16.153183 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152313 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:40:16.153183 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152315 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:40:16.153183 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152318 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:40:16.153183 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152322 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:40:16.153183 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152325 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:40:16.153183 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152328 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:40:16.153183 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152330 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:40:16.153183 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152333 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:40:16.153183 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152336 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:40:16.153690 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152338 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:40:16.153690 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152341 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:40:16.153690 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152343 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:40:16.153690 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152346 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:40:16.153690 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152348 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:40:16.153690 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152351 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:40:16.153690 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152353 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:40:16.153690 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152356 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:40:16.153690 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152359 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:40:16.153690 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152362 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:40:16.153690 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152365 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:40:16.153690 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152367 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:40:16.153690 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152373 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:40:16.153690 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152375 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:40:16.153690 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152378 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:40:16.153690 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152380 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:40:16.153690 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152383 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:40:16.153690 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152387 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:40:16.153690 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152391 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:40:16.154152 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152394 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:40:16.154152 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152396 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:40:16.154152 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152399 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:40:16.154152 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152402 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:40:16.154152 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152404 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:40:16.154152 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152407 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:40:16.154152 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152409 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:40:16.154152 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152413 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:40:16.154152 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152416 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:40:16.154152 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152418 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:40:16.154152 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152421 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:40:16.154152 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152424 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:40:16.154152 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152426 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:40:16.154152 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152429 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:40:16.154152 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152432 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:40:16.154152 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152434 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:40:16.154152 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152436 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:40:16.154152 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152439 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:40:16.154152 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152441 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:40:16.154152 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152831 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:40:16.154662 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152836 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:40:16.154662 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152839 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:40:16.154662 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152842 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:40:16.154662 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152844 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:40:16.154662 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152848 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:40:16.154662 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152851 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:40:16.154662 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152853 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:40:16.154662 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152856 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:40:16.154662 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152859 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:40:16.154662 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152862 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:40:16.154662 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152865 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:40:16.154662 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152867 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:40:16.154662 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152870 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:40:16.154662 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152872 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:40:16.154662 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152875 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:40:16.154662 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152878 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:40:16.154662 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152880 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:40:16.154662 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152882 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:40:16.154662 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152885 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:40:16.154662 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152889 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:40:16.155149 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152892 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:40:16.155149 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152894 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:40:16.155149 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152897 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:40:16.155149 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152899 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:40:16.155149 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152902 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:40:16.155149 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152905 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:40:16.155149 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152907 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:40:16.155149 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152909 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:40:16.155149 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152912 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:40:16.155149 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152914 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:40:16.155149 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152917 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:40:16.155149 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152920 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:40:16.155149 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152922 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:40:16.155149 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152925 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:40:16.155149 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152927 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:40:16.155149 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152930 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:40:16.155149 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152932 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:40:16.155149 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152935 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:40:16.155149 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152938 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:40:16.155149 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152940 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:40:16.155626 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152942 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:40:16.155626 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152945 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:40:16.155626 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152948 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:40:16.155626 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152950 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:40:16.155626 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152953 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:40:16.155626 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152956 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:40:16.155626 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152958 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:40:16.155626 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152961 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:40:16.155626 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152964 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:40:16.155626 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152967 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:40:16.155626 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152969 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:40:16.155626 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152972 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:40:16.155626 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152974 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:40:16.155626 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152977 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:40:16.155626 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152980 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:40:16.155626 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152982 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:40:16.155626 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152985 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:40:16.155626 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152988 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:40:16.155626 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152990 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:40:16.155626 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152993 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:40:16.156153 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152995 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:40:16.156153 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.152998 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:40:16.156153 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153000 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:40:16.156153 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153003 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:40:16.156153 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153006 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:40:16.156153 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153009 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:40:16.156153 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153011 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:40:16.156153 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153014 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:40:16.156153 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153016 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:40:16.156153 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153019 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:40:16.156153 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153021 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:40:16.156153 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153026 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:40:16.156153 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153030 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:40:16.156153 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153034 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:40:16.156153 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153038 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:40:16.156153 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153041 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:40:16.156153 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153044 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:40:16.156153 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153047 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:40:16.156153 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153049 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:40:16.156615 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153052 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:40:16.156615 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153055 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:40:16.156615 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153057 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:40:16.156615 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153060 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:40:16.156615 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153062 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:40:16.156615 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153065 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:40:16.156615 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153067 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:40:16.156615 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153069 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:40:16.156615 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153072 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:40:16.156615 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153074 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:40:16.156615 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153077 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:40:16.156615 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153079 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:40:16.156615 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153082 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:40:16.156615 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153084 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:40:16.156615 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153088 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:40:16.156615 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153091 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:40:16.156615 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153095 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:40:16.156615 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153097 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:40:16.156615 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153100 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153192 2557 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153209 2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153215 2557 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153220 2557 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153224 2557 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153228 2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153232 2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153236 2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153240 2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153243 2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153247 2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153250 2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153253 2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153257 2557 flags.go:64] FLAG: --cgroup-root="" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153259 2557 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153262 2557 flags.go:64] FLAG: --client-ca-file="" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153265 2557 flags.go:64] FLAG: --cloud-provider="external" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153268 2557 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153273 2557 flags.go:64] FLAG: --cluster-domain="" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153276 2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153279 2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153282 2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153285 2557 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 15:40:16.157098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153289 2557 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153292 2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153295 2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153298 2557 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153301 2557 flags.go:64] FLAG: --contention-profiling="false" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153305 2557 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153308 2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153313 2557 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153317 2557 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153321 2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153324 2557 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153327 2557 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153330 2557 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153332 2557 flags.go:64] FLAG: --enable-server="true" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153335 2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153340 2557 flags.go:64] FLAG: --event-burst="100" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153343 2557 flags.go:64] FLAG: --event-qps="50" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153346 2557 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153349 2557 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153352 2557 flags.go:64] FLAG: --eviction-hard="" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153356 2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153359 2557 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153362 2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153365 2557 flags.go:64] FLAG: --eviction-soft="" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153368 2557 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 15:40:16.157740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153371 2557 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153374 2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153377 2557 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153380 2557 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153383 2557 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153386 2557 flags.go:64] FLAG: --feature-gates="" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153389 2557 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153392 2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153395 2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153399 2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153402 2557 flags.go:64] FLAG: --healthz-port="10248" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153405 2557 flags.go:64] FLAG: --help="false" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153408 2557 flags.go:64] FLAG: --hostname-override="ip-10-0-139-248.ec2.internal" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153411 2557 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153415 2557 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153418 2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153421 2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153425 2557 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153428 2557 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153431 2557 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153434 2557 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153437 2557 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153440 2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153443 2557 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 15:40:16.158399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153447 2557 flags.go:64] FLAG: --kube-reserved="" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153449 2557 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153452 2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153455 2557 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153458 2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153461 2557 flags.go:64] FLAG: --lock-file="" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153464 2557 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153476 2557 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153480 2557 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153486 2557 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153489 2557 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153492 2557 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153495 2557 flags.go:64] FLAG: --logging-format="text" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153498 2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153501 2557 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153504 2557 flags.go:64] FLAG: --manifest-url="" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153507 2557 flags.go:64] FLAG: --manifest-url-header="" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153511 2557 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153520 2557 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153525 2557 flags.go:64] FLAG: --max-pods="110" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153528 2557 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153531 2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153536 2557 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153539 2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153542 2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 15:40:16.159016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153544 2557 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153547 2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153555 2557 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153560 2557 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153563 2557 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153566 2557 flags.go:64] FLAG: --pod-cidr="" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153569 2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153575 2557 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153579 2557 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153582 2557 flags.go:64] FLAG: --pods-per-core="0" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153585 2557 flags.go:64] FLAG: --port="10250" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153588 2557 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153591 2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ace168e5f57500ee" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153594 2557 flags.go:64] FLAG: --qos-reserved="" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153597 2557 flags.go:64] FLAG: --read-only-port="10255" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153600 2557 flags.go:64] FLAG: --register-node="true" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153603 2557 flags.go:64] FLAG: --register-with-taints="" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153607 2557 flags.go:64] FLAG: --registry-burst="10" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153610 2557 flags.go:64] FLAG: --registry-qps="5" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153613 2557 flags.go:64] FLAG: --reserved-cpus="" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153615 2557 flags.go:64] FLAG: --reserved-memory="" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153619 2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153623 2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153626 2557 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153629 2557 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 15:40:16.159643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153632 2557 flags.go:64] FLAG: --runonce="false" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153635 2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153639 2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153642 2557 flags.go:64] FLAG: --seccomp-default="false" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153646 2557 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153649 2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153652 2557 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153655 2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153658 2557 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153661 2557 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153663 2557 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153668 2557 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153671 2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153674 2557 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153677 2557 flags.go:64] FLAG: --system-cgroups="" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153680 2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153685 2557 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153688 2557 flags.go:64] FLAG: --tls-cert-file="" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153692 2557 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153699 2557 flags.go:64] FLAG: --tls-min-version="" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153702 2557 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153705 2557 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153708 2557 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153711 2557 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153714 2557 flags.go:64] FLAG: --v="2" Jul 07 15:40:16.160298 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153718 2557 flags.go:64] FLAG: --version="false" Jul 07 15:40:16.160892 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153722 2557 flags.go:64] FLAG: --vmodule="" Jul 07 15:40:16.160892 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153726 2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 15:40:16.160892 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.153729 2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 15:40:16.160892 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153849 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:40:16.160892 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153853 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:40:16.160892 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153856 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:40:16.160892 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153859 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:40:16.160892 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153861 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:40:16.160892 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153864 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:40:16.160892 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153867 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:40:16.160892 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153873 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:40:16.160892 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153876 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:40:16.160892 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153878 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:40:16.160892 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153881 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:40:16.160892 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153884 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:40:16.160892 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153886 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:40:16.160892 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153889 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:40:16.160892 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153893 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:40:16.160892 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153896 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:40:16.160892 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153898 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:40:16.161689 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153901 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:40:16.161689 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153904 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:40:16.161689 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153906 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:40:16.161689 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153910 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:40:16.161689 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153912 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:40:16.161689 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153915 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:40:16.161689 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153917 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:40:16.161689 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153920 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:40:16.161689 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153922 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:40:16.161689 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153925 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:40:16.161689 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153929 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:40:16.161689 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153932 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:40:16.161689 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153935 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:40:16.161689 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153938 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:40:16.161689 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153941 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:40:16.161689 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153944 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:40:16.161689 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153947 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:40:16.161689 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153949 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:40:16.161689 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153952 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:40:16.161689 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153954 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:40:16.162433 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153957 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:40:16.162433 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153959 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:40:16.162433 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153963 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:40:16.162433 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153966 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:40:16.162433 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153968 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:40:16.162433 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153971 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:40:16.162433 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153975 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:40:16.162433 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153978 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:40:16.162433 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153981 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:40:16.162433 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153987 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:40:16.162433 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153990 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:40:16.162433 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153993 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:40:16.162433 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153996 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:40:16.162433 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.153999 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:40:16.162433 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154002 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:40:16.162433 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154006 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:40:16.162433 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154010 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:40:16.162433 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154012 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:40:16.162433 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154015 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:40:16.163032 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154018 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:40:16.163032 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154021 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:40:16.163032 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154023 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:40:16.163032 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154026 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:40:16.163032 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154029 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:40:16.163032 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154032 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:40:16.163032 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154035 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:40:16.163032 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154037 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:40:16.163032 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154040 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:40:16.163032 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154043 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:40:16.163032 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154045 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:40:16.163032 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154048 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:40:16.163032 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154051 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:40:16.163032 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154053 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:40:16.163032 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154056 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:40:16.163032 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154059 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:40:16.163032 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154062 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:40:16.163032 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154065 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:40:16.163032 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154067 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:40:16.163032 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154070 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:40:16.163550 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154072 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:40:16.163550 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154075 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:40:16.163550 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154079 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:40:16.163550 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154082 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:40:16.163550 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154085 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:40:16.163550 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154087 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:40:16.163550 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154090 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:40:16.163550 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154092 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:40:16.163550 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154094 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:40:16.163550 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154097 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:40:16.163550 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154100 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:40:16.163550 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154102 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:40:16.163550 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154105 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:40:16.163550 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154108 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:40:16.163550 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154126 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:40:16.163550 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154128 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:40:16.163550 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154131 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:40:16.163550 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154134 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:40:16.163550 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154137 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:40:16.163550 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154141 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:40:16.164033 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154144 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:40:16.164033 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154147 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:40:16.164033 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.154150 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:40:16.164033 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.154155 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 15:40:16.164033 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.163412 2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 15:40:16.164033 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.163429 2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 15:40:16.164302 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.164292 2557 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 15:40:16.164338 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.164302 2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 15:40:16.164383 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164375 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:40:16.164383 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164383 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:40:16.164434 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164386 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:40:16.164434 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164389 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:40:16.164434 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164392 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:40:16.164434 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164395 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:40:16.164434 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164398 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:40:16.164434 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164401 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:40:16.164434 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164404 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:40:16.164434 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164407 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:40:16.164434 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164410 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:40:16.164434 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164413 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:40:16.164434 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164415 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:40:16.164434 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164418 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:40:16.164434 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164421 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:40:16.164434 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164424 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:40:16.164434 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164427 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:40:16.164434 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164429 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:40:16.164434 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164431 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:40:16.164434 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164434 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:40:16.164434 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164437 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:40:16.164434 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164440 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:40:16.164910 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164443 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:40:16.164910 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164446 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:40:16.164910 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164449 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:40:16.164910 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164453 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:40:16.164910 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164456 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:40:16.164910 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164460 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:40:16.164910 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164462 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:40:16.164910 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164465 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:40:16.164910 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164467 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:40:16.164910 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164470 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:40:16.164910 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164473 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:40:16.164910 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164475 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:40:16.164910 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164478 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:40:16.164910 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164481 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:40:16.164910 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164483 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:40:16.164910 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164486 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:40:16.164910 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164488 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:40:16.164910 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164491 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:40:16.164910 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164493 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:40:16.165415 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164496 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:40:16.165415 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164498 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:40:16.165415 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164502 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:40:16.165415 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164506 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:40:16.165415 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164509 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:40:16.165415 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164512 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:40:16.165415 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164515 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:40:16.165415 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164518 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:40:16.165415 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164520 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:40:16.165415 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164523 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:40:16.165415 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164525 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:40:16.165415 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164528 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:40:16.165415 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164531 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:40:16.165415 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164533 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:40:16.165415 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164537 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:40:16.165415 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164540 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:40:16.165415 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164543 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:40:16.165415 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164545 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:40:16.165415 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164548 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:40:16.165415 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164550 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:40:16.165901 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164553 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:40:16.165901 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164556 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:40:16.165901 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164558 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:40:16.165901 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164561 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:40:16.165901 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164564 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:40:16.165901 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164566 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:40:16.165901 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164569 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:40:16.165901 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164571 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:40:16.165901 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164574 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:40:16.165901 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164576 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:40:16.165901 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164579 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:40:16.165901 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164581 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:40:16.165901 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164584 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:40:16.165901 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164587 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:40:16.165901 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164589 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:40:16.165901 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164592 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:40:16.165901 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164594 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:40:16.165901 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164596 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:40:16.165901 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164600 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:40:16.165901 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164602 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:40:16.166387 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164605 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:40:16.166387 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164608 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:40:16.166387 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164610 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:40:16.166387 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164613 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:40:16.166387 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164615 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:40:16.166387 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164618 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:40:16.166387 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164623 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:40:16.166387 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164626 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:40:16.166387 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164629 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:40:16.166387 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164632 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:40:16.166387 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164635 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:40:16.166387 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164638 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:40:16.166387 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164641 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:40:16.166387 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164643 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:40:16.166387 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164646 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:40:16.166387 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164649 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:40:16.166387 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164651 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:40:16.166387 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164654 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:40:16.166808 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.164659 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 15:40:16.166808 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164788 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:40:16.166808 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164793 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:40:16.166808 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164796 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:40:16.166808 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164798 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:40:16.166808 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164801 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:40:16.166808 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164804 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:40:16.166808 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164806 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:40:16.166808 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164809 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:40:16.166808 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164812 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:40:16.166808 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164815 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:40:16.166808 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164817 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:40:16.166808 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164820 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:40:16.166808 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164824 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:40:16.166808 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164827 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:40:16.167204 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164831 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:40:16.167204 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164833 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:40:16.167204 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164836 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:40:16.167204 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164838 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:40:16.167204 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164841 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:40:16.167204 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164844 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:40:16.167204 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164846 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:40:16.167204 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164849 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:40:16.167204 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164852 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:40:16.167204 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164854 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:40:16.167204 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164857 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:40:16.167204 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164860 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:40:16.167204 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164863 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:40:16.167204 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164865 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:40:16.167204 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164868 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:40:16.167204 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164870 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:40:16.167204 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164873 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:40:16.167204 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164876 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:40:16.167204 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164878 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:40:16.167204 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164881 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:40:16.167682 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164883 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:40:16.167682 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164886 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:40:16.167682 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164889 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:40:16.167682 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164891 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:40:16.167682 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164894 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:40:16.167682 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164896 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:40:16.167682 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164899 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:40:16.167682 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164902 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:40:16.167682 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164904 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:40:16.167682 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164907 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:40:16.167682 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164912 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:40:16.167682 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164916 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:40:16.167682 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164919 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:40:16.167682 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164922 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:40:16.167682 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164925 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:40:16.167682 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164928 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:40:16.167682 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164931 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:40:16.167682 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164934 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:40:16.167682 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164936 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:40:16.167682 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164939 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:40:16.168349 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164942 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:40:16.168349 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164945 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:40:16.168349 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164949 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:40:16.168349 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164951 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:40:16.168349 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164954 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:40:16.168349 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164957 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:40:16.168349 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164960 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:40:16.168349 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164962 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:40:16.168349 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164965 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:40:16.168349 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164967 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:40:16.168349 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164970 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:40:16.168349 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164972 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:40:16.168349 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164975 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:40:16.168349 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164977 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:40:16.168349 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164980 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:40:16.168349 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164982 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:40:16.168349 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164985 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:40:16.168349 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164988 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:40:16.168349 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164990 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:40:16.168812 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164993 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:40:16.168812 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164995 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:40:16.168812 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.164998 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:40:16.168812 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165001 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:40:16.168812 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165004 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:40:16.168812 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165007 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:40:16.168812 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165010 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:40:16.168812 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165012 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:40:16.168812 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165015 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:40:16.168812 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165017 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:40:16.168812 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165020 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:40:16.168812 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165023 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:40:16.168812 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165025 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:40:16.168812 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165028 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:40:16.168812 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165030 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:40:16.168812 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165033 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:40:16.168812 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165035 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:40:16.168812 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165038 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:40:16.168812 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165041 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:40:16.168812 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165043 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:40:16.169315 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165046 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:40:16.169315 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165048 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:40:16.169315 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165051 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:40:16.169315 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165053 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:40:16.169315 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165056 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:40:16.169315 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:16.165058 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:40:16.169315 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.165063 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 15:40:16.169315 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.165198 2557 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 15:40:16.169315 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.167816 2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 15:40:16.169555 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.169426 2557 server.go:1019] "Starting client certificate rotation" Jul 07 15:40:16.169555 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.169533 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 15:40:16.169611 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.169575 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 15:40:16.172213 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.172189 2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 15:40:16.177101 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.177067 2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 15:40:16.188500 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.188481 2557 log.go:25] "Validated CRI v1 runtime API" Jul 07 15:40:16.191413 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.191391 2557 log.go:25] "Validated CRI v1 image API" Jul 07 15:40:16.192389 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.192370 2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 15:40:16.195244 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.195210 2557 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ad16dbb2-2877-4747-8727-9ce5073a7317:/dev/nvme0n1p4 b103d310-1a89-49a4-b978-7bdaf93952bd:/dev/nvme0n1p3] Jul 07 15:40:16.195344 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.195244 2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 15:40:16.201376 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.201349 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 15:40:16.201531 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.201261 2557 manager.go:216] Machine: {Timestamp:2026-07-07 15:40:16.200684901 +0000 UTC m=+0.378329361 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104960 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27e49bb8f77fdd2cb5debd3ee61a43 SystemUUID:ec27e49b-b8f7-7fdd-2cb5-debd3ee61a43 BootID:b4d0f257-b3f5-43ab-bb11-cf3b1144ec48 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6e:1b:03:71:5d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6e:1b:03:71:5d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:07:0b:bf:93:48 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 15:40:16.201531 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.201447 2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 15:40:16.201687 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.201666 2557 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 15:40:16.204518 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.204484 2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 15:40:16.204667 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.204519 2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-248.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 15:40:16.205412 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.205400 2557 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 15:40:16.205448 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.205414 2557 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 15:40:16.205448 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.205422 2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 15:40:16.205448 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.205437 2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 15:40:16.207503 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.207490 2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 15:40:16.212136 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.212106 2557 state_mem.go:36] "Initialized new in-memory state store" Jul 07 15:40:16.212311 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.212300 2557 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 15:40:16.212349 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.212336 2557 kubelet.go:486] "Attempting to sync node with API server" Jul 07 15:40:16.212379 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.212350 2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 15:40:16.212379 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.212373 2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 15:40:16.212437 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.212385 2557 kubelet.go:398] "Adding apiserver pod source" Jul 07 15:40:16.212437 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.212398 2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 15:40:16.212844 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.212828 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 15:40:16.212896 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.212856 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 15:40:16.215448 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.215422 2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 15:40:16.218089 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.218074 2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 15:40:16.218158 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.218107 2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 15:40:16.219574 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.219556 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 15:40:16.219574 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.219576 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 15:40:16.219746 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.219582 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 15:40:16.219746 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.219588 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 15:40:16.219746 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.219597 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 15:40:16.219746 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.219602 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 15:40:16.219746 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.219612 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 15:40:16.219746 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.219617 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 15:40:16.219746 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.219624 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 15:40:16.219746 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.219630 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 15:40:16.219746 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.219638 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 15:40:16.219746 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.219645 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 15:40:16.220694 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.220679 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 15:40:16.220756 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.220698 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 15:40:16.223940 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.223926 2557 server.go:1268] "Started kubelet" Jul 07 15:40:16.224156 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.224085 2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 15:40:16.224216 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.224186 2557 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 15:40:16.224728 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.224699 2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 15:40:16.224800 ip-10-0-139-248 systemd[1]: Started Kubernetes Kubelet. Jul 07 15:40:16.226443 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.226396 2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 15:40:16.226852 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.226835 2557 server.go:310] "Adding debug handlers to kubelet server" Jul 07 15:40:16.229775 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:16.229744 2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 15:40:16.231243 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.231219 2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-248.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 15:40:16.232070 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:16.232048 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 15:40:16.232145 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:16.232054 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-248.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 15:40:16.232302 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:16.231154 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-248.ec2.internal.18c00c277fab1fe0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-248.ec2.internal,UID:ip-10-0-139-248.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-248.ec2.internal,},FirstTimestamp:2026-07-07 15:40:16.22389552 +0000 UTC m=+0.401539960,LastTimestamp:2026-07-07 15:40:16.22389552 +0000 UTC m=+0.401539960,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-248.ec2.internal,}" Jul 07 15:40:16.235091 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.235077 2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 15:40:16.235162 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.235089 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 15:40:16.235615 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:16.235243 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found" Jul 07 15:40:16.236299 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.236284 2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 15:40:16.236856 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.236296 2557 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 15:40:16.236856 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.236420 2557 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 15:40:16.236856 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.236573 2557 reconstruct.go:97] "Volume reconstruction finished" Jul 07 15:40:16.236856 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.236582 2557 reconciler.go:29] "Reconciler: start to sync state" Jul 07 15:40:16.236856 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.236725 2557 factory.go:55] Registering systemd factory Jul 07 15:40:16.236856 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.236741 2557 factory.go:223] Registration of the systemd container factory successfully Jul 07 15:40:16.237078 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.237020 2557 factory.go:153] Registering CRI-O factory Jul 07 15:40:16.237078 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.237033 2557 factory.go:223] Registration of the crio container factory successfully Jul 07 15:40:16.237173 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.237089 2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 15:40:16.237173 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.237138 2557 factory.go:103] Registering Raw factory Jul 07 15:40:16.237173 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.237154 2557 manager.go:1196] Started watching for new ooms in manager Jul 07 15:40:16.237586 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:16.237565 2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-248.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 15:40:16.237738 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:16.237719 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 15:40:16.237818 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.237806 2557 manager.go:349] Starting recovery of all containers Jul 07 15:40:16.247654 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.247626 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vfh8x" Jul 07 15:40:16.250644 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.250530 2557 manager.go:354] Recovery completed Jul 07 15:40:16.254664 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.254651 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:40:16.255045 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.255025 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vfh8x" Jul 07 15:40:16.256719 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.256700 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:40:16.256789 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.256735 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:40:16.256789 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.256750 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:40:16.257331 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.257313 2557 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 15:40:16.257331 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.257330 2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 15:40:16.257462 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.257336 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:40:16.257462 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.257349 2557 state_mem.go:36] "Initialized new in-memory state store" Jul 07 15:40:16.258282 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.258266 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:40:16.258336 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.258295 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:40:16.258336 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.258309 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:40:16.259364 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.259351 2557 policy_none.go:49] "None policy: Start" Jul 07 15:40:16.259423 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.259367 2557 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 15:40:16.259423 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.259377 2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 15:40:16.261273 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.261264 2557 policy_none.go:47] "Start" Jul 07 15:40:16.301816 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.301797 2557 manager.go:337] "Starting Device Plugin manager" Jul 07 15:40:16.316211 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:16.301840 2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 15:40:16.316211 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.301853 2557 server.go:87] "Starting device plugin registration server" Jul 07 15:40:16.316211 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.302215 2557 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 15:40:16.316211 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.302239 2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 15:40:16.316211 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.302324 2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 15:40:16.316211 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.302413 2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 15:40:16.316211 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.302421 2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 15:40:16.316211 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:16.303270 2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 15:40:16.316211 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:16.303322 2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-248.ec2.internal\" not found" Jul 07 15:40:16.342776 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.342737 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 15:40:16.344073 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.344052 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 15:40:16.344216 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.344098 2557 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 15:40:16.344216 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.344136 2557 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 15:40:16.344216 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:16.344200 2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 15:40:16.346822 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.346801 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 15:40:16.402906 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.402827 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:40:16.404107 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.404088 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:40:16.404196 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.404141 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:40:16.404196 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.404157 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:40:16.404196 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.404189 2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-248.ec2.internal" Jul 07 15:40:16.413845 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.413822 2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-248.ec2.internal" Jul 07 15:40:16.413912 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:16.413852 2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-248.ec2.internal\": node \"ip-10-0-139-248.ec2.internal\" not found" Jul 07 15:40:16.437679 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:16.437647 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found" Jul 07 15:40:16.444529 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.444505 2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal"] Jul 07 15:40:16.444589 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.444570 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:40:16.446144 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.446102 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:40:16.446266 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.446154 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:40:16.446266 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.446165 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:40:16.448562 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.448549 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:40:16.448693 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.448677 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal" Jul 07 15:40:16.448735 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.448707 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:40:16.449364 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.449346 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:40:16.449460 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.449378 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:40:16.449460 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.449392 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:40:16.449460 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.449397 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:40:16.449460 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.449416 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:40:16.449460 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.449432 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:40:16.452267 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.452250 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal" Jul 07 15:40:16.452355 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.452278 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:40:16.453068 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.453049 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:40:16.453168 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.453078 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:40:16.453168 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.453091 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:40:16.474860 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:16.474833 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-248.ec2.internal\" not found" node="ip-10-0-139-248.ec2.internal" Jul 07 15:40:16.479028 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:16.479011 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-248.ec2.internal\" not found" node="ip-10-0-139-248.ec2.internal" Jul 07 15:40:16.537597 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.537562 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d0a6e48ccbac6ae57118c8e5af923f7b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-248.ec2.internal\" (UID: \"d0a6e48ccbac6ae57118c8e5af923f7b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal" Jul 07 15:40:16.537597 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.537597 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f1f4c5f2ef966b7794d2c0682814a647-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal\" (UID: \"f1f4c5f2ef966b7794d2c0682814a647\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal" Jul 07 15:40:16.537792 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.537617 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1f4c5f2ef966b7794d2c0682814a647-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal\" (UID: \"f1f4c5f2ef966b7794d2c0682814a647\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal" Jul 07 15:40:16.538606 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:16.538580 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found" Jul 07 15:40:16.638071 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.638032 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d0a6e48ccbac6ae57118c8e5af923f7b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-248.ec2.internal\" (UID: \"d0a6e48ccbac6ae57118c8e5af923f7b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal" Jul 07 15:40:16.638071 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.638076 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f1f4c5f2ef966b7794d2c0682814a647-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal\" (UID: \"f1f4c5f2ef966b7794d2c0682814a647\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal" Jul 07 15:40:16.638281 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.638097 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1f4c5f2ef966b7794d2c0682814a647-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal\" (UID: \"f1f4c5f2ef966b7794d2c0682814a647\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal" Jul 07 15:40:16.638281 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.638146 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d0a6e48ccbac6ae57118c8e5af923f7b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-248.ec2.internal\" (UID: \"d0a6e48ccbac6ae57118c8e5af923f7b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal" Jul 07 15:40:16.638281 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.638150 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1f4c5f2ef966b7794d2c0682814a647-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal\" (UID: \"f1f4c5f2ef966b7794d2c0682814a647\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal" Jul 07 15:40:16.638281 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.638151 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f1f4c5f2ef966b7794d2c0682814a647-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal\" (UID: \"f1f4c5f2ef966b7794d2c0682814a647\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal" Jul 07 15:40:16.639084 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:16.639059 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found" Jul 07 15:40:16.739910 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:16.739825 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found" Jul 07 15:40:16.777965 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.777933 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal" Jul 07 15:40:16.782498 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:16.782475 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal" Jul 07 15:40:16.840950 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:16.840901 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found" Jul 07 15:40:16.942462 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:16.942418 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found" Jul 07 15:40:17.042667 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:17.042576 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found" Jul 07 15:40:17.143344 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:17.143316 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found" Jul 07 15:40:17.169471 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:17.169425 2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 15:40:17.169683 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:17.169661 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 15:40:17.224635 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:17.224603 2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 15:40:17.244325 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:17.244288 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found" Jul 07 15:40:17.258525 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:17.258472 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 15:35:16 +0000 UTC" deadline="2028-03-31 22:14:52.216324988 +0000 UTC" Jul 07 15:40:17.258525 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:17.258524 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15198h34m34.957804285s" Jul 07 15:40:17.344678 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:17.344591 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found" Jul 07 15:40:17.359976 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:17.359938 2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 15:40:17.435752 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:17.435714 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal" Jul 07 15:40:17.450412 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:17.450381 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 15:40:17.452067 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:17.452038 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal" Jul 07 15:40:17.461723 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:17.461703 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 15:40:17.522727 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:17.522678 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a6e48ccbac6ae57118c8e5af923f7b.slice/crio-a26edcf5d47b5f4d7fc3acf1947fe8dfef599b90ada51246ee903757fba9b69c WatchSource:0}: Error finding container a26edcf5d47b5f4d7fc3acf1947fe8dfef599b90ada51246ee903757fba9b69c: Status 404 returned error can't find the container with id a26edcf5d47b5f4d7fc3acf1947fe8dfef599b90ada51246ee903757fba9b69c Jul 07 15:40:17.527829 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:17.527808 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 15:40:17.675776 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:17.675751 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f4c5f2ef966b7794d2c0682814a647.slice/crio-1c3bf37c671a74d88c31c3fd3f3840d1cc6f90ce7931e826450c8727d1c071b1 WatchSource:0}: Error finding container 1c3bf37c671a74d88c31c3fd3f3840d1cc6f90ce7931e826450c8727d1c071b1: Status 404 returned error can't find the container with id 1c3bf37c671a74d88c31c3fd3f3840d1cc6f90ce7931e826450c8727d1c071b1 Jul 07 15:40:17.837150 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:17.837109 2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 15:40:18.215179 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.215137 2557 apiserver.go:52] "Watching apiserver" Jul 07 15:40:18.219895 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.219867 2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 15:40:18.221801 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.221773 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-mcrlt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal","openshift-multus/multus-additional-cni-plugins-2l5gl","openshift-multus/multus-f46rg","openshift-multus/network-metrics-daemon-tb5cw","kube-system/global-pull-secret-syncer-q2m8b","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6","openshift-dns/node-resolver-lhvjn","openshift-network-diagnostics/network-check-target-hwwk7","openshift-network-operator/iptables-alerter-q7l26","openshift-ovn-kubernetes/ovnkube-node-hkj2z","kube-system/konnectivity-agent-vktnc","kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal","openshift-cluster-node-tuning-operator/tuned-8r9n4"] Jul 07 15:40:18.224783 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.224758 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.226889 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.226859 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 15:40:18.227011 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.226861 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 15:40:18.227011 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.226959 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.227121 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.227027 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 15:40:18.227197 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.226960 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gx9k2\"" Jul 07 15:40:18.228747 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.228728 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 15:40:18.228852 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.228769 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8p26p\"" Jul 07 15:40:18.229012 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.228998 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 15:40:18.229335 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.229320 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.231132 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.231113 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 15:40:18.231277 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.231259 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 15:40:18.231382 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.231363 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 15:40:18.231449 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.231402 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kzc4f\"" Jul 07 15:40:18.231449 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.231408 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 15:40:18.231786 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.231770 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mcrlt" Jul 07 15:40:18.233473 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.233453 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 15:40:18.233573 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.233533 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 15:40:18.233573 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.233536 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 15:40:18.233573 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.233561 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lk648\"" Jul 07 15:40:18.234219 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.234200 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:18.234305 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:18.234280 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5cw" podUID="285707f9-4f1b-480e-ae8e-123d5ba608c6" Jul 07 15:40:18.235322 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.235304 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 15:40:18.236462 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.236446 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:18.236539 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:18.236506 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2m8b" podUID="b99c24f7-0333-47fe-8b1a-ed9c6a10d23c" Jul 07 15:40:18.238921 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.238895 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.240822 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.240802 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 15:40:18.240966 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.240947 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 15:40:18.241107 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.241089 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-p5w4m\"" Jul 07 15:40:18.241301 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.241284 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lhvjn" Jul 07 15:40:18.242859 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.242840 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 15:40:18.243284 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.243258 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6s466\"" Jul 07 15:40:18.243704 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.243516 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 15:40:18.245633 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.245036 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:18.245633 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:18.245258 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hwwk7" podUID="a767f075-0967-445e-8038-d22a7eda5b15" Jul 07 15:40:18.246796 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.246777 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 15:40:18.247817 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.247794 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q7l26" Jul 07 15:40:18.249510 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.249479 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6660812-8311-4ffc-8966-bb7324e3afbb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jwfx6\" (UID: \"c6660812-8311-4ffc-8966-bb7324e3afbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.249620 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.249514 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d4917d4-c3ba-432c-83b2-0672de4a70f8-system-cni-dir\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.249620 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.249540 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-run\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.249620 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.249566 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-kubelet-config\") pod \"global-pull-secret-syncer-q2m8b\" (UID: \"b99c24f7-0333-47fe-8b1a-ed9c6a10d23c\") " pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:18.249620 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.249589 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-dbus\") pod \"global-pull-secret-syncer-q2m8b\" (UID: \"b99c24f7-0333-47fe-8b1a-ed9c6a10d23c\") " pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:18.249620 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.249618 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-os-release\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.249890 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.249666 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78c8109c-5327-468b-ba1d-327ab10afd8e-cni-binary-copy\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.249890 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.249700 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-542m8\" (UniqueName: \"kubernetes.io/projected/78c8109c-5327-468b-ba1d-327ab10afd8e-kube-api-access-542m8\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.249890 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.249757 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d4917d4-c3ba-432c-83b2-0672de4a70f8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.249890 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.249785 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwcgl\" (UniqueName: \"kubernetes.io/projected/7d4917d4-c3ba-432c-83b2-0672de4a70f8-kube-api-access-jwcgl\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.249890 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.249808 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-cnibin\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.249890 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.249841 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 15:40:18.249890 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.249851 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-system-cni-dir\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.249890 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.249873 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-host-var-lib-cni-bin\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.249890 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.249891 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-host-run-multus-certs\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.250410 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.249917 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-etc-kubernetes\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.250410 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.249937 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7d4917d4-c3ba-432c-83b2-0672de4a70f8-os-release\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.250410 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.249995 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d4917d4-c3ba-432c-83b2-0672de4a70f8-cni-binary-copy\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.250410 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250043 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-sys\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.250410 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250063 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g5qgl\"" Jul 07 15:40:18.250410 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250063 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-host-var-lib-cni-multus\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.250410 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250096 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6660812-8311-4ffc-8966-bb7324e3afbb-socket-dir\") pod \"aws-ebs-csi-driver-node-jwfx6\" (UID: \"c6660812-8311-4ffc-8966-bb7324e3afbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.250410 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250134 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6660812-8311-4ffc-8966-bb7324e3afbb-etc-selinux\") pod \"aws-ebs-csi-driver-node-jwfx6\" (UID: \"c6660812-8311-4ffc-8966-bb7324e3afbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.250410 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250062 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 15:40:18.250410 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250156 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-host\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.250410 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250196 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-multus-socket-dir-parent\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.250410 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250214 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-host-run-k8s-cni-cncf-io\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.250410 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250216 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 15:40:18.250410 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250229 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-host-var-lib-kubelet\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.250410 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250251 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-etc-systemd\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.250410 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250270 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztgfr\" (UniqueName: \"kubernetes.io/projected/c6660812-8311-4ffc-8966-bb7324e3afbb-kube-api-access-ztgfr\") pod \"aws-ebs-csi-driver-node-jwfx6\" (UID: \"c6660812-8311-4ffc-8966-bb7324e3afbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.250410 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250302 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d4917d4-c3ba-432c-83b2-0672de4a70f8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.250410 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250324 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-etc-sysctl-conf\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.251287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250338 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/78c8109c-5327-468b-ba1d-327ab10afd8e-multus-daemon-config\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.251287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250358 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdjwd\" (UniqueName: \"kubernetes.io/projected/285707f9-4f1b-480e-ae8e-123d5ba608c6-kube-api-access-zdjwd\") pod \"network-metrics-daemon-tb5cw\" (UID: \"285707f9-4f1b-480e-ae8e-123d5ba608c6\") " pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:18.251287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250380 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-etc-sysctl-d\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.251287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250407 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bbsn\" (UniqueName: \"kubernetes.io/projected/acd15283-0678-4182-928a-cfd5f57c2923-kube-api-access-8bbsn\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.251287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250423 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-hostroot\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.251287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250446 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-multus-conf-dir\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.251287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250480 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d4917d4-c3ba-432c-83b2-0672de4a70f8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.251287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250499 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-etc-kubernetes\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.251287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250520 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/acd15283-0678-4182-928a-cfd5f57c2923-etc-tuned\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.251287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250548 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6660812-8311-4ffc-8966-bb7324e3afbb-sys-fs\") pod \"aws-ebs-csi-driver-node-jwfx6\" (UID: \"c6660812-8311-4ffc-8966-bb7324e3afbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.251287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250564 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d4917d4-c3ba-432c-83b2-0672de4a70f8-cnibin\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.251287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250579 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acd15283-0678-4182-928a-cfd5f57c2923-tmp\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.251287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250599 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/78ecda82-54b3-4727-89b9-5d55dd0ca428-serviceca\") pod \"node-ca-mcrlt\" (UID: \"78ecda82-54b3-4727-89b9-5d55dd0ca428\") " pod="openshift-image-registry/node-ca-mcrlt" Jul 07 15:40:18.251287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250637 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret\") pod \"global-pull-secret-syncer-q2m8b\" (UID: \"b99c24f7-0333-47fe-8b1a-ed9c6a10d23c\") " pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:18.251287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250660 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-etc-modprobe-d\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.251287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250682 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78ecda82-54b3-4727-89b9-5d55dd0ca428-host\") pod \"node-ca-mcrlt\" (UID: \"78ecda82-54b3-4727-89b9-5d55dd0ca428\") " pod="openshift-image-registry/node-ca-mcrlt" Jul 07 15:40:18.251287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250687 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.252103 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250709 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6660812-8311-4ffc-8966-bb7324e3afbb-registration-dir\") pod \"aws-ebs-csi-driver-node-jwfx6\" (UID: \"c6660812-8311-4ffc-8966-bb7324e3afbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.252103 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250733 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-lib-modules\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.252103 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250759 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs\") pod \"network-metrics-daemon-tb5cw\" (UID: \"285707f9-4f1b-480e-ae8e-123d5ba608c6\") " pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:18.252103 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250786 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6660812-8311-4ffc-8966-bb7324e3afbb-device-dir\") pod \"aws-ebs-csi-driver-node-jwfx6\" (UID: \"c6660812-8311-4ffc-8966-bb7324e3afbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.252103 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250810 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4md8b\" (UniqueName: \"kubernetes.io/projected/78ecda82-54b3-4727-89b9-5d55dd0ca428-kube-api-access-4md8b\") pod \"node-ca-mcrlt\" (UID: \"78ecda82-54b3-4727-89b9-5d55dd0ca428\") " pod="openshift-image-registry/node-ca-mcrlt" Jul 07 15:40:18.252103 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250833 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-etc-sysconfig\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.252103 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250856 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-var-lib-kubelet\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.252103 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250880 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-multus-cni-dir\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.252103 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.250903 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-host-run-netns\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.252613 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.252596 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f67x4\"" Jul 07 15:40:18.252680 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.252654 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 15:40:18.252913 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.252894 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 15:40:18.252987 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.252897 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 15:40:18.253047 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.253032 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 15:40:18.253142 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.253124 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 15:40:18.253228 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.253172 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 15:40:18.253466 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.253447 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vktnc" Jul 07 15:40:18.255417 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.255373 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rtzwd\"" Jul 07 15:40:18.255663 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.255641 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 15:40:18.256072 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.255861 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 15:40:18.274644 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.274621 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zpvfd" Jul 07 15:40:18.283673 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.283645 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zpvfd" Jul 07 15:40:18.338863 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.338831 2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 15:40:18.349464 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.349393 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal" event={"ID":"f1f4c5f2ef966b7794d2c0682814a647","Type":"ContainerStarted","Data":"1c3bf37c671a74d88c31c3fd3f3840d1cc6f90ce7931e826450c8727d1c071b1"} Jul 07 15:40:18.350949 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.350916 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal" event={"ID":"d0a6e48ccbac6ae57118c8e5af923f7b","Type":"ContainerStarted","Data":"a26edcf5d47b5f4d7fc3acf1947fe8dfef599b90ada51246ee903757fba9b69c"} Jul 07 15:40:18.351115 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351096 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-kubelet-config\") pod \"global-pull-secret-syncer-q2m8b\" (UID: \"b99c24f7-0333-47fe-8b1a-ed9c6a10d23c\") " pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:18.351191 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351125 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-dbus\") pod \"global-pull-secret-syncer-q2m8b\" (UID: \"b99c24f7-0333-47fe-8b1a-ed9c6a10d23c\") " pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:18.351191 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351171 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-os-release\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.351295 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351199 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78c8109c-5327-468b-ba1d-327ab10afd8e-cni-binary-copy\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.351295 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351220 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-kubelet-config\") pod \"global-pull-secret-syncer-q2m8b\" (UID: \"b99c24f7-0333-47fe-8b1a-ed9c6a10d23c\") " pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:18.351295 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351224 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-542m8\" (UniqueName: \"kubernetes.io/projected/78c8109c-5327-468b-ba1d-327ab10afd8e-kube-api-access-542m8\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.351295 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351274 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-systemd-units\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.351537 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351306 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d4917d4-c3ba-432c-83b2-0672de4a70f8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.351537 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351324 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-os-release\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.351537 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351338 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwcgl\" (UniqueName: \"kubernetes.io/projected/7d4917d4-c3ba-432c-83b2-0672de4a70f8-kube-api-access-jwcgl\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.351537 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351365 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-cnibin\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.351537 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351389 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/07a80436-65b9-4f0d-99b3-ecda36240b48-ovnkube-script-lib\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.351537 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351415 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-system-cni-dir\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.351537 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351442 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-host-var-lib-cni-bin\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.351537 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351467 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-host-run-multus-certs\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.351537 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351503 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0de85e66-128c-44a2-b566-5770f8b132e0-iptables-alerter-script\") pod \"iptables-alerter-q7l26\" (UID: \"0de85e66-128c-44a2-b566-5770f8b132e0\") " pod="openshift-network-operator/iptables-alerter-q7l26" Jul 07 15:40:18.351537 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351529 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3567278c-a1c9-403d-9353-80da294fafb9-agent-certs\") pod \"konnectivity-agent-vktnc\" (UID: \"3567278c-a1c9-403d-9353-80da294fafb9\") " pod="kube-system/konnectivity-agent-vktnc" Jul 07 15:40:18.351537 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351538 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-host-var-lib-cni-bin\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.352101 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351555 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-etc-kubernetes\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.352101 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351580 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7d4917d4-c3ba-432c-83b2-0672de4a70f8-os-release\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.352101 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351597 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-system-cni-dir\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.352101 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351608 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d4917d4-c3ba-432c-83b2-0672de4a70f8-cni-binary-copy\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.352101 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351634 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-sys\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.352101 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351658 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-host-run-multus-certs\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.352101 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351660 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-host-var-lib-cni-multus\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.352101 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351470 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-dbus\") pod \"global-pull-secret-syncer-q2m8b\" (UID: \"b99c24f7-0333-47fe-8b1a-ed9c6a10d23c\") " pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:18.352101 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351706 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-host-var-lib-cni-multus\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.352101 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351736 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7b29c32-fe48-437a-956a-f4b0d4eab6c3-tmp-dir\") pod \"node-resolver-lhvjn\" (UID: \"e7b29c32-fe48-437a-956a-f4b0d4eab6c3\") " pod="openshift-dns/node-resolver-lhvjn" Jul 07 15:40:18.352101 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351743 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7d4917d4-c3ba-432c-83b2-0672de4a70f8-os-release\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.352101 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351762 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-run-systemd\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.352101 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351787 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-node-log\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.352101 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351796 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-etc-kubernetes\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.352101 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351816 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6660812-8311-4ffc-8966-bb7324e3afbb-socket-dir\") pod \"aws-ebs-csi-driver-node-jwfx6\" (UID: \"c6660812-8311-4ffc-8966-bb7324e3afbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.352101 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351824 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-cnibin\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.352101 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351844 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6660812-8311-4ffc-8966-bb7324e3afbb-etc-selinux\") pod \"aws-ebs-csi-driver-node-jwfx6\" (UID: \"c6660812-8311-4ffc-8966-bb7324e3afbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.352101 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351853 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78c8109c-5327-468b-ba1d-327ab10afd8e-cni-binary-copy\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.353029 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351872 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-host\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.353029 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351904 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-multus-socket-dir-parent\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.353029 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351876 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-sys\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.353029 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351924 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-host\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.353029 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351931 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-host-run-k8s-cni-cncf-io\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.353029 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351961 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-host-var-lib-kubelet\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.353029 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351961 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d4917d4-c3ba-432c-83b2-0672de4a70f8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.353029 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.351986 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-host-run-netns\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.353029 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352014 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6660812-8311-4ffc-8966-bb7324e3afbb-etc-selinux\") pod \"aws-ebs-csi-driver-node-jwfx6\" (UID: \"c6660812-8311-4ffc-8966-bb7324e3afbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.353029 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352019 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-host-cni-bin\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.353029 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352035 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-host-run-k8s-cni-cncf-io\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.353029 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352020 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6660812-8311-4ffc-8966-bb7324e3afbb-socket-dir\") pod \"aws-ebs-csi-driver-node-jwfx6\" (UID: \"c6660812-8311-4ffc-8966-bb7324e3afbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.353029 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352062 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-host-var-lib-kubelet\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.353029 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352093 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-etc-systemd\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.353029 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352114 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e7b29c32-fe48-437a-956a-f4b0d4eab6c3-hosts-file\") pod \"node-resolver-lhvjn\" (UID: \"e7b29c32-fe48-437a-956a-f4b0d4eab6c3\") " pod="openshift-dns/node-resolver-lhvjn" Jul 07 15:40:18.353029 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352143 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-run-ovn\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.353029 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352163 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-etc-systemd\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.353766 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352168 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/07a80436-65b9-4f0d-99b3-ecda36240b48-env-overrides\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.353766 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352205 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/07a80436-65b9-4f0d-99b3-ecda36240b48-ovn-node-metrics-cert\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.353766 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352245 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-multus-socket-dir-parent\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.353766 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352254 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3567278c-a1c9-403d-9353-80da294fafb9-konnectivity-ca\") pod \"konnectivity-agent-vktnc\" (UID: \"3567278c-a1c9-403d-9353-80da294fafb9\") " pod="kube-system/konnectivity-agent-vktnc" Jul 07 15:40:18.353766 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352290 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztgfr\" (UniqueName: \"kubernetes.io/projected/c6660812-8311-4ffc-8966-bb7324e3afbb-kube-api-access-ztgfr\") pod \"aws-ebs-csi-driver-node-jwfx6\" (UID: \"c6660812-8311-4ffc-8966-bb7324e3afbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.353766 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352298 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d4917d4-c3ba-432c-83b2-0672de4a70f8-cni-binary-copy\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.353766 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352330 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d4917d4-c3ba-432c-83b2-0672de4a70f8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.353766 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352360 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-etc-sysctl-conf\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.353766 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352537 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/78c8109c-5327-468b-ba1d-327ab10afd8e-multus-daemon-config\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.353766 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352551 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d4917d4-c3ba-432c-83b2-0672de4a70f8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.353766 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352569 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdjwd\" (UniqueName: \"kubernetes.io/projected/285707f9-4f1b-480e-ae8e-123d5ba608c6-kube-api-access-zdjwd\") pod \"network-metrics-daemon-tb5cw\" (UID: \"285707f9-4f1b-480e-ae8e-123d5ba608c6\") " pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:18.353766 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352599 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.353766 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352623 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-etc-sysctl-d\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.353766 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352644 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-host-slash\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.353766 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352668 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/07a80436-65b9-4f0d-99b3-ecda36240b48-ovnkube-config\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.353766 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352688 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-etc-sysctl-conf\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.354579 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352693 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tztcg\" (UniqueName: \"kubernetes.io/projected/0de85e66-128c-44a2-b566-5770f8b132e0-kube-api-access-tztcg\") pod \"iptables-alerter-q7l26\" (UID: \"0de85e66-128c-44a2-b566-5770f8b132e0\") " pod="openshift-network-operator/iptables-alerter-q7l26" Jul 07 15:40:18.354579 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352806 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-etc-sysctl-d\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.354579 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352837 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbsn\" (UniqueName: \"kubernetes.io/projected/acd15283-0678-4182-928a-cfd5f57c2923-kube-api-access-8bbsn\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.354579 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352865 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-hostroot\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.354579 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352889 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-multus-conf-dir\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.354579 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352916 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-host-kubelet\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.354579 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352942 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-host-run-ovn-kubernetes\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.354579 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352969 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d4917d4-c3ba-432c-83b2-0672de4a70f8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.354579 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352974 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-hostroot\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.354579 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.352995 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-etc-kubernetes\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.354579 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353019 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-multus-conf-dir\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.354579 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353124 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/78c8109c-5327-468b-ba1d-327ab10afd8e-multus-daemon-config\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.354579 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353153 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/acd15283-0678-4182-928a-cfd5f57c2923-etc-tuned\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.354579 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353199 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-host-cni-netd\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.354579 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353221 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0de85e66-128c-44a2-b566-5770f8b132e0-host-slash\") pod \"iptables-alerter-q7l26\" (UID: \"0de85e66-128c-44a2-b566-5770f8b132e0\") " pod="openshift-network-operator/iptables-alerter-q7l26" Jul 07 15:40:18.354579 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353236 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-etc-kubernetes\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.354579 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353268 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6660812-8311-4ffc-8966-bb7324e3afbb-sys-fs\") pod \"aws-ebs-csi-driver-node-jwfx6\" (UID: \"c6660812-8311-4ffc-8966-bb7324e3afbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.355483 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353304 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d4917d4-c3ba-432c-83b2-0672de4a70f8-cnibin\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.355483 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353328 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acd15283-0678-4182-928a-cfd5f57c2923-tmp\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.355483 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353357 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/78ecda82-54b3-4727-89b9-5d55dd0ca428-serviceca\") pod \"node-ca-mcrlt\" (UID: \"78ecda82-54b3-4727-89b9-5d55dd0ca428\") " pod="openshift-image-registry/node-ca-mcrlt" Jul 07 15:40:18.355483 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353368 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d4917d4-c3ba-432c-83b2-0672de4a70f8-cnibin\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.355483 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353398 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret\") pod \"global-pull-secret-syncer-q2m8b\" (UID: \"b99c24f7-0333-47fe-8b1a-ed9c6a10d23c\") " pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:18.355483 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353426 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-etc-modprobe-d\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.355483 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353450 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78ecda82-54b3-4727-89b9-5d55dd0ca428-host\") pod \"node-ca-mcrlt\" (UID: \"78ecda82-54b3-4727-89b9-5d55dd0ca428\") " pod="openshift-image-registry/node-ca-mcrlt" Jul 07 15:40:18.355483 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353480 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-var-lib-openvswitch\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.355483 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353505 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-run-openvswitch\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.355483 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353528 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-log-socket\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.355483 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353546 2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 15:40:18.355483 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353614 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6660812-8311-4ffc-8966-bb7324e3afbb-registration-dir\") pod \"aws-ebs-csi-driver-node-jwfx6\" (UID: \"c6660812-8311-4ffc-8966-bb7324e3afbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.355483 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353621 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78ecda82-54b3-4727-89b9-5d55dd0ca428-host\") pod \"node-ca-mcrlt\" (UID: \"78ecda82-54b3-4727-89b9-5d55dd0ca428\") " pod="openshift-image-registry/node-ca-mcrlt" Jul 07 15:40:18.355483 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353555 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6660812-8311-4ffc-8966-bb7324e3afbb-registration-dir\") pod \"aws-ebs-csi-driver-node-jwfx6\" (UID: \"c6660812-8311-4ffc-8966-bb7324e3afbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.355483 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353613 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-etc-modprobe-d\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.355483 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353656 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-lib-modules\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.355483 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353682 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs\") pod \"network-metrics-daemon-tb5cw\" (UID: \"285707f9-4f1b-480e-ae8e-123d5ba608c6\") " pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:18.356343 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353711 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmr97\" (UniqueName: \"kubernetes.io/projected/e7b29c32-fe48-437a-956a-f4b0d4eab6c3-kube-api-access-nmr97\") pod \"node-resolver-lhvjn\" (UID: \"e7b29c32-fe48-437a-956a-f4b0d4eab6c3\") " pod="openshift-dns/node-resolver-lhvjn" Jul 07 15:40:18.356343 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:18.353714 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:18.356343 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353367 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6660812-8311-4ffc-8966-bb7324e3afbb-sys-fs\") pod \"aws-ebs-csi-driver-node-jwfx6\" (UID: \"c6660812-8311-4ffc-8966-bb7324e3afbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.356343 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:18.353774 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:18.356343 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353786 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-lib-modules\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.356343 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:18.353798 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret podName:b99c24f7-0333-47fe-8b1a-ed9c6a10d23c nodeName:}" failed. No retries permitted until 2026-07-07 15:40:18.853780951 +0000 UTC m=+3.031425383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret") pod "global-pull-secret-syncer-q2m8b" (UID: "b99c24f7-0333-47fe-8b1a-ed9c6a10d23c") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:18.356343 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353834 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/78ecda82-54b3-4727-89b9-5d55dd0ca428-serviceca\") pod \"node-ca-mcrlt\" (UID: \"78ecda82-54b3-4727-89b9-5d55dd0ca428\") " pod="openshift-image-registry/node-ca-mcrlt" Jul 07 15:40:18.356343 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:18.353840 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs podName:285707f9-4f1b-480e-ae8e-123d5ba608c6 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:18.853822911 +0000 UTC m=+3.031467350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs") pod "network-metrics-daemon-tb5cw" (UID: "285707f9-4f1b-480e-ae8e-123d5ba608c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:18.356343 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353880 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-etc-openvswitch\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.356343 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353911 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6660812-8311-4ffc-8966-bb7324e3afbb-device-dir\") pod \"aws-ebs-csi-driver-node-jwfx6\" (UID: \"c6660812-8311-4ffc-8966-bb7324e3afbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.356343 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353952 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d4917d4-c3ba-432c-83b2-0672de4a70f8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.356343 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.353959 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6660812-8311-4ffc-8966-bb7324e3afbb-device-dir\") pod \"aws-ebs-csi-driver-node-jwfx6\" (UID: \"c6660812-8311-4ffc-8966-bb7324e3afbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.356343 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.354013 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4md8b\" (UniqueName: \"kubernetes.io/projected/78ecda82-54b3-4727-89b9-5d55dd0ca428-kube-api-access-4md8b\") pod \"node-ca-mcrlt\" (UID: \"78ecda82-54b3-4727-89b9-5d55dd0ca428\") " pod="openshift-image-registry/node-ca-mcrlt" Jul 07 15:40:18.356343 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.354057 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-etc-sysconfig\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.356343 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.354085 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-var-lib-kubelet\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.356343 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.354164 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-etc-sysconfig\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.356343 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.354212 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-multus-cni-dir\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.357306 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.354279 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-var-lib-kubelet\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.357306 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.354307 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-multus-cni-dir\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.357306 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.354237 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-host-run-netns\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.357306 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.354363 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2klg\" (UniqueName: \"kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg\") pod \"network-check-target-hwwk7\" (UID: \"a767f075-0967-445e-8038-d22a7eda5b15\") " pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:18.357306 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.354413 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6660812-8311-4ffc-8966-bb7324e3afbb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jwfx6\" (UID: \"c6660812-8311-4ffc-8966-bb7324e3afbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.357306 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.354487 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/78c8109c-5327-468b-ba1d-327ab10afd8e-host-run-netns\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.357306 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.354517 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6660812-8311-4ffc-8966-bb7324e3afbb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jwfx6\" (UID: \"c6660812-8311-4ffc-8966-bb7324e3afbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.357306 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.354440 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d4917d4-c3ba-432c-83b2-0672de4a70f8-system-cni-dir\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.357306 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.354601 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-run\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.357306 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.354606 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d4917d4-c3ba-432c-83b2-0672de4a70f8-system-cni-dir\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.357306 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.354633 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95ncb\" (UniqueName: \"kubernetes.io/projected/07a80436-65b9-4f0d-99b3-ecda36240b48-kube-api-access-95ncb\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.357306 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.354681 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/acd15283-0678-4182-928a-cfd5f57c2923-run\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.357902 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.357645 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/acd15283-0678-4182-928a-cfd5f57c2923-etc-tuned\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.357991 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.357964 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acd15283-0678-4182-928a-cfd5f57c2923-tmp\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.360856 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.360826 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwcgl\" (UniqueName: \"kubernetes.io/projected/7d4917d4-c3ba-432c-83b2-0672de4a70f8-kube-api-access-jwcgl\") pod \"multus-additional-cni-plugins-2l5gl\" (UID: \"7d4917d4-c3ba-432c-83b2-0672de4a70f8\") " pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.361081 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.361030 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bbsn\" (UniqueName: \"kubernetes.io/projected/acd15283-0678-4182-928a-cfd5f57c2923-kube-api-access-8bbsn\") pod \"tuned-8r9n4\" (UID: \"acd15283-0678-4182-928a-cfd5f57c2923\") " pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.361539 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.361286 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztgfr\" (UniqueName: \"kubernetes.io/projected/c6660812-8311-4ffc-8966-bb7324e3afbb-kube-api-access-ztgfr\") pod \"aws-ebs-csi-driver-node-jwfx6\" (UID: \"c6660812-8311-4ffc-8966-bb7324e3afbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.361539 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.361319 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-542m8\" (UniqueName: \"kubernetes.io/projected/78c8109c-5327-468b-ba1d-327ab10afd8e-kube-api-access-542m8\") pod \"multus-f46rg\" (UID: \"78c8109c-5327-468b-ba1d-327ab10afd8e\") " pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.361539 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.361473 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdjwd\" (UniqueName: \"kubernetes.io/projected/285707f9-4f1b-480e-ae8e-123d5ba608c6-kube-api-access-zdjwd\") pod \"network-metrics-daemon-tb5cw\" (UID: \"285707f9-4f1b-480e-ae8e-123d5ba608c6\") " pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:18.362967 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.362947 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4md8b\" (UniqueName: \"kubernetes.io/projected/78ecda82-54b3-4727-89b9-5d55dd0ca428-kube-api-access-4md8b\") pod \"node-ca-mcrlt\" (UID: \"78ecda82-54b3-4727-89b9-5d55dd0ca428\") " pod="openshift-image-registry/node-ca-mcrlt" Jul 07 15:40:18.455910 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.455867 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmr97\" (UniqueName: \"kubernetes.io/projected/e7b29c32-fe48-437a-956a-f4b0d4eab6c3-kube-api-access-nmr97\") pod \"node-resolver-lhvjn\" (UID: \"e7b29c32-fe48-437a-956a-f4b0d4eab6c3\") " pod="openshift-dns/node-resolver-lhvjn" Jul 07 15:40:18.455910 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.455918 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-etc-openvswitch\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.456227 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.455950 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2klg\" (UniqueName: \"kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg\") pod \"network-check-target-hwwk7\" (UID: \"a767f075-0967-445e-8038-d22a7eda5b15\") " pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:18.456227 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.455979 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95ncb\" (UniqueName: \"kubernetes.io/projected/07a80436-65b9-4f0d-99b3-ecda36240b48-kube-api-access-95ncb\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.456227 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456008 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-systemd-units\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.456227 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456017 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-etc-openvswitch\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.456227 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456055 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/07a80436-65b9-4f0d-99b3-ecda36240b48-ovnkube-script-lib\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.456227 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456128 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-systemd-units\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.456227 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456164 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0de85e66-128c-44a2-b566-5770f8b132e0-iptables-alerter-script\") pod \"iptables-alerter-q7l26\" (UID: \"0de85e66-128c-44a2-b566-5770f8b132e0\") " pod="openshift-network-operator/iptables-alerter-q7l26" Jul 07 15:40:18.456227 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456229 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3567278c-a1c9-403d-9353-80da294fafb9-agent-certs\") pod \"konnectivity-agent-vktnc\" (UID: \"3567278c-a1c9-403d-9353-80da294fafb9\") " pod="kube-system/konnectivity-agent-vktnc" Jul 07 15:40:18.456636 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456266 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7b29c32-fe48-437a-956a-f4b0d4eab6c3-tmp-dir\") pod \"node-resolver-lhvjn\" (UID: \"e7b29c32-fe48-437a-956a-f4b0d4eab6c3\") " pod="openshift-dns/node-resolver-lhvjn" Jul 07 15:40:18.456636 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456292 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-run-systemd\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.456636 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456321 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-node-log\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.456636 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456354 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-host-run-netns\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.456636 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456380 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-host-cni-bin\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.456636 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456405 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e7b29c32-fe48-437a-956a-f4b0d4eab6c3-hosts-file\") pod \"node-resolver-lhvjn\" (UID: \"e7b29c32-fe48-437a-956a-f4b0d4eab6c3\") " pod="openshift-dns/node-resolver-lhvjn" Jul 07 15:40:18.456636 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456430 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-run-ovn\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.456636 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456420 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-node-log\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.456636 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456462 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/07a80436-65b9-4f0d-99b3-ecda36240b48-env-overrides\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.456636 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456491 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/07a80436-65b9-4f0d-99b3-ecda36240b48-ovn-node-metrics-cert\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.456636 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456498 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-run-systemd\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.456636 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456516 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3567278c-a1c9-403d-9353-80da294fafb9-konnectivity-ca\") pod \"konnectivity-agent-vktnc\" (UID: \"3567278c-a1c9-403d-9353-80da294fafb9\") " pod="kube-system/konnectivity-agent-vktnc" Jul 07 15:40:18.456636 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456549 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.456636 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456579 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-host-slash\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.456636 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456603 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/07a80436-65b9-4f0d-99b3-ecda36240b48-ovnkube-config\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.456636 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456621 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e7b29c32-fe48-437a-956a-f4b0d4eab6c3-hosts-file\") pod \"node-resolver-lhvjn\" (UID: \"e7b29c32-fe48-437a-956a-f4b0d4eab6c3\") " pod="openshift-dns/node-resolver-lhvjn" Jul 07 15:40:18.456636 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456627 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tztcg\" (UniqueName: \"kubernetes.io/projected/0de85e66-128c-44a2-b566-5770f8b132e0-kube-api-access-tztcg\") pod \"iptables-alerter-q7l26\" (UID: \"0de85e66-128c-44a2-b566-5770f8b132e0\") " pod="openshift-network-operator/iptables-alerter-q7l26" Jul 07 15:40:18.457445 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456652 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7b29c32-fe48-437a-956a-f4b0d4eab6c3-tmp-dir\") pod \"node-resolver-lhvjn\" (UID: \"e7b29c32-fe48-437a-956a-f4b0d4eab6c3\") " pod="openshift-dns/node-resolver-lhvjn" Jul 07 15:40:18.457445 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456660 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-host-kubelet\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.457445 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456686 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-host-run-ovn-kubernetes\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.457445 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456666 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-host-run-netns\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.457445 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456716 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-host-cni-netd\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.457445 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456725 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-host-slash\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.457445 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456742 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0de85e66-128c-44a2-b566-5770f8b132e0-host-slash\") pod \"iptables-alerter-q7l26\" (UID: \"0de85e66-128c-44a2-b566-5770f8b132e0\") " pod="openshift-network-operator/iptables-alerter-q7l26" Jul 07 15:40:18.457445 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456747 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-host-cni-bin\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.457445 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456846 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0de85e66-128c-44a2-b566-5770f8b132e0-iptables-alerter-script\") pod \"iptables-alerter-q7l26\" (UID: \"0de85e66-128c-44a2-b566-5770f8b132e0\") " pod="openshift-network-operator/iptables-alerter-q7l26" Jul 07 15:40:18.457445 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456846 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/07a80436-65b9-4f0d-99b3-ecda36240b48-ovnkube-script-lib\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.457445 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456881 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.457445 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456891 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0de85e66-128c-44a2-b566-5770f8b132e0-host-slash\") pod \"iptables-alerter-q7l26\" (UID: \"0de85e66-128c-44a2-b566-5770f8b132e0\") " pod="openshift-network-operator/iptables-alerter-q7l26" Jul 07 15:40:18.457445 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456908 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-host-run-ovn-kubernetes\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.457445 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456926 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-host-cni-netd\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.457445 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456951 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-host-kubelet\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.457445 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456968 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-var-lib-openvswitch\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.457445 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.456998 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-run-openvswitch\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.458106 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.457006 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/07a80436-65b9-4f0d-99b3-ecda36240b48-env-overrides\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.458106 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.457050 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-run-ovn\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.458106 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.457036 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-log-socket\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.458106 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.457071 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-log-socket\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.458106 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.457093 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-run-openvswitch\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.458106 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.457111 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07a80436-65b9-4f0d-99b3-ecda36240b48-var-lib-openvswitch\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.458106 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.457438 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/07a80436-65b9-4f0d-99b3-ecda36240b48-ovnkube-config\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.458106 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.457496 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3567278c-a1c9-403d-9353-80da294fafb9-konnectivity-ca\") pod \"konnectivity-agent-vktnc\" (UID: \"3567278c-a1c9-403d-9353-80da294fafb9\") " pod="kube-system/konnectivity-agent-vktnc" Jul 07 15:40:18.459519 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.459494 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/07a80436-65b9-4f0d-99b3-ecda36240b48-ovn-node-metrics-cert\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.459701 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.459682 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3567278c-a1c9-403d-9353-80da294fafb9-agent-certs\") pod \"konnectivity-agent-vktnc\" (UID: \"3567278c-a1c9-403d-9353-80da294fafb9\") " pod="kube-system/konnectivity-agent-vktnc" Jul 07 15:40:18.463338 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:18.463310 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:40:18.463338 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:18.463341 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:40:18.463544 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:18.463355 2557 projected.go:196] Error preparing data for projected volume kube-api-access-m2klg for pod openshift-network-diagnostics/network-check-target-hwwk7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:18.463544 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:18.463429 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg podName:a767f075-0967-445e-8038-d22a7eda5b15 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:18.963408451 +0000 UTC m=+3.141052907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m2klg" (UniqueName: "kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg") pod "network-check-target-hwwk7" (UID: "a767f075-0967-445e-8038-d22a7eda5b15") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:18.466155 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.466092 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmr97\" (UniqueName: \"kubernetes.io/projected/e7b29c32-fe48-437a-956a-f4b0d4eab6c3-kube-api-access-nmr97\") pod \"node-resolver-lhvjn\" (UID: \"e7b29c32-fe48-437a-956a-f4b0d4eab6c3\") " pod="openshift-dns/node-resolver-lhvjn" Jul 07 15:40:18.466332 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.466310 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95ncb\" (UniqueName: \"kubernetes.io/projected/07a80436-65b9-4f0d-99b3-ecda36240b48-kube-api-access-95ncb\") pod \"ovnkube-node-hkj2z\" (UID: \"07a80436-65b9-4f0d-99b3-ecda36240b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.466978 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.466955 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tztcg\" (UniqueName: \"kubernetes.io/projected/0de85e66-128c-44a2-b566-5770f8b132e0-kube-api-access-tztcg\") pod \"iptables-alerter-q7l26\" (UID: \"0de85e66-128c-44a2-b566-5770f8b132e0\") " pod="openshift-network-operator/iptables-alerter-q7l26" Jul 07 15:40:18.469979 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.469958 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 15:40:18.536855 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.536816 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" Jul 07 15:40:18.541749 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.541715 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" Jul 07 15:40:18.551264 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:18.551222 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6660812_8311_4ffc_8966_bb7324e3afbb.slice/crio-6a2d241e7fc2e64e7a25547287e23be244b38cc67c3221507ad4c9db5b93ddcd WatchSource:0}: Error finding container 6a2d241e7fc2e64e7a25547287e23be244b38cc67c3221507ad4c9db5b93ddcd: Status 404 returned error can't find the container with id 6a2d241e7fc2e64e7a25547287e23be244b38cc67c3221507ad4c9db5b93ddcd Jul 07 15:40:18.552679 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.552647 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f46rg" Jul 07 15:40:18.555421 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:18.555382 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd15283_0678_4182_928a_cfd5f57c2923.slice/crio-41b4775e81e54c6b655cdf783bc0030668fbba74c6a8e5ecad1e0ff68e6874a8 WatchSource:0}: Error finding container 41b4775e81e54c6b655cdf783bc0030668fbba74c6a8e5ecad1e0ff68e6874a8: Status 404 returned error can't find the container with id 41b4775e81e54c6b655cdf783bc0030668fbba74c6a8e5ecad1e0ff68e6874a8 Jul 07 15:40:18.559274 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.559252 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mcrlt" Jul 07 15:40:18.565257 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:18.565224 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c8109c_5327_468b_ba1d_327ab10afd8e.slice/crio-61756becf858becb5814a3820a8ef2bcf425c7c72a5ab5414dd0acc10836b959 WatchSource:0}: Error finding container 61756becf858becb5814a3820a8ef2bcf425c7c72a5ab5414dd0acc10836b959: Status 404 returned error can't find the container with id 61756becf858becb5814a3820a8ef2bcf425c7c72a5ab5414dd0acc10836b959 Jul 07 15:40:18.566396 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.566327 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2l5gl" Jul 07 15:40:18.570229 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:18.570176 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ecda82_54b3_4727_89b9_5d55dd0ca428.slice/crio-22d271ec3ba41c122e81383d1a7383e8bb36df54f976e142ae72c26f7f0b70a0 WatchSource:0}: Error finding container 22d271ec3ba41c122e81383d1a7383e8bb36df54f976e142ae72c26f7f0b70a0: Status 404 returned error can't find the container with id 22d271ec3ba41c122e81383d1a7383e8bb36df54f976e142ae72c26f7f0b70a0 Jul 07 15:40:18.575818 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.575792 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lhvjn" Jul 07 15:40:18.577036 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:18.577007 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4917d4_c3ba_432c_83b2_0672de4a70f8.slice/crio-07cdcb67bb92dfaff50cdab623aa03baea4b6b2ef46b85b39714c5c70dafda0a WatchSource:0}: Error finding container 07cdcb67bb92dfaff50cdab623aa03baea4b6b2ef46b85b39714c5c70dafda0a: Status 404 returned error can't find the container with id 07cdcb67bb92dfaff50cdab623aa03baea4b6b2ef46b85b39714c5c70dafda0a Jul 07 15:40:18.585665 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.585632 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q7l26" Jul 07 15:40:18.587716 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:18.587686 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b29c32_fe48_437a_956a_f4b0d4eab6c3.slice/crio-4b4fbf5662face00fdf242372db790dec94c34e041c1a72f52612f68e5068a52 WatchSource:0}: Error finding container 4b4fbf5662face00fdf242372db790dec94c34e041c1a72f52612f68e5068a52: Status 404 returned error can't find the container with id 4b4fbf5662face00fdf242372db790dec94c34e041c1a72f52612f68e5068a52 Jul 07 15:40:18.592466 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.592438 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:18.597752 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:18.597709 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de85e66_128c_44a2_b566_5770f8b132e0.slice/crio-635ac4b83e3e36f675278535003b83b690689ca5697ff61d7ddcbba76eb2d39e WatchSource:0}: Error finding container 635ac4b83e3e36f675278535003b83b690689ca5697ff61d7ddcbba76eb2d39e: Status 404 returned error can't find the container with id 635ac4b83e3e36f675278535003b83b690689ca5697ff61d7ddcbba76eb2d39e Jul 07 15:40:18.598948 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.598925 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vktnc" Jul 07 15:40:18.604649 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:18.604616 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a80436_65b9_4f0d_99b3_ecda36240b48.slice/crio-960c5668319b2dc56f217d9558075e2c2944890e46d743c11299b4be61f61382 WatchSource:0}: Error finding container 960c5668319b2dc56f217d9558075e2c2944890e46d743c11299b4be61f61382: Status 404 returned error can't find the container with id 960c5668319b2dc56f217d9558075e2c2944890e46d743c11299b4be61f61382 Jul 07 15:40:18.611052 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:18.611026 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3567278c_a1c9_403d_9353_80da294fafb9.slice/crio-1211100c900dca3562d69b783310350e0751d4dc925a32c65f027f4d8e0c38a8 WatchSource:0}: Error finding container 1211100c900dca3562d69b783310350e0751d4dc925a32c65f027f4d8e0c38a8: Status 404 returned error can't find the container with id 1211100c900dca3562d69b783310350e0751d4dc925a32c65f027f4d8e0c38a8 Jul 07 15:40:18.860584 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.860492 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret\") pod \"global-pull-secret-syncer-q2m8b\" (UID: \"b99c24f7-0333-47fe-8b1a-ed9c6a10d23c\") " pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:18.860584 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:18.860548 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs\") pod \"network-metrics-daemon-tb5cw\" (UID: \"285707f9-4f1b-480e-ae8e-123d5ba608c6\") " pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:18.860817 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:18.860694 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:18.860817 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:18.860722 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:18.860817 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:18.860786 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret podName:b99c24f7-0333-47fe-8b1a-ed9c6a10d23c nodeName:}" failed. No retries permitted until 2026-07-07 15:40:19.860763508 +0000 UTC m=+4.038407957 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret") pod "global-pull-secret-syncer-q2m8b" (UID: "b99c24f7-0333-47fe-8b1a-ed9c6a10d23c") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:18.860817 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:18.860806 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs podName:285707f9-4f1b-480e-ae8e-123d5ba608c6 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:19.860797138 +0000 UTC m=+4.038441568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs") pod "network-metrics-daemon-tb5cw" (UID: "285707f9-4f1b-480e-ae8e-123d5ba608c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:19.061746 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:19.061711 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2klg\" (UniqueName: \"kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg\") pod \"network-check-target-hwwk7\" (UID: \"a767f075-0967-445e-8038-d22a7eda5b15\") " pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:19.061934 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:19.061898 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:40:19.061934 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:19.061922 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:40:19.061934 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:19.061933 2557 projected.go:196] Error preparing data for projected volume kube-api-access-m2klg for pod openshift-network-diagnostics/network-check-target-hwwk7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:19.062065 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:19.061991 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg podName:a767f075-0967-445e-8038-d22a7eda5b15 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:20.061975577 +0000 UTC m=+4.239620006 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m2klg" (UniqueName: "kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg") pod "network-check-target-hwwk7" (UID: "a767f075-0967-445e-8038-d22a7eda5b15") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:19.288474 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:19.288117 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 15:35:18 +0000 UTC" deadline="2028-01-28 06:54:23.314783815 +0000 UTC" Jul 07 15:40:19.288474 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:19.288419 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13671h14m4.026370007s" Jul 07 15:40:19.357526 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:19.357480 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f46rg" event={"ID":"78c8109c-5327-468b-ba1d-327ab10afd8e","Type":"ContainerStarted","Data":"61756becf858becb5814a3820a8ef2bcf425c7c72a5ab5414dd0acc10836b959"} Jul 07 15:40:19.360368 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:19.360326 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" event={"ID":"acd15283-0678-4182-928a-cfd5f57c2923","Type":"ContainerStarted","Data":"41b4775e81e54c6b655cdf783bc0030668fbba74c6a8e5ecad1e0ff68e6874a8"} Jul 07 15:40:19.365936 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:19.365891 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" event={"ID":"c6660812-8311-4ffc-8966-bb7324e3afbb","Type":"ContainerStarted","Data":"6a2d241e7fc2e64e7a25547287e23be244b38cc67c3221507ad4c9db5b93ddcd"} Jul 07 15:40:19.379543 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:19.379468 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" event={"ID":"07a80436-65b9-4f0d-99b3-ecda36240b48","Type":"ContainerStarted","Data":"960c5668319b2dc56f217d9558075e2c2944890e46d743c11299b4be61f61382"} Jul 07 15:40:19.386232 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:19.384374 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q7l26" event={"ID":"0de85e66-128c-44a2-b566-5770f8b132e0","Type":"ContainerStarted","Data":"635ac4b83e3e36f675278535003b83b690689ca5697ff61d7ddcbba76eb2d39e"} Jul 07 15:40:19.389507 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:19.389468 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mcrlt" event={"ID":"78ecda82-54b3-4727-89b9-5d55dd0ca428","Type":"ContainerStarted","Data":"22d271ec3ba41c122e81383d1a7383e8bb36df54f976e142ae72c26f7f0b70a0"} Jul 07 15:40:19.397287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:19.397251 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vktnc" event={"ID":"3567278c-a1c9-403d-9353-80da294fafb9","Type":"ContainerStarted","Data":"1211100c900dca3562d69b783310350e0751d4dc925a32c65f027f4d8e0c38a8"} Jul 07 15:40:19.406966 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:19.406927 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lhvjn" event={"ID":"e7b29c32-fe48-437a-956a-f4b0d4eab6c3","Type":"ContainerStarted","Data":"4b4fbf5662face00fdf242372db790dec94c34e041c1a72f52612f68e5068a52"} Jul 07 15:40:19.412355 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:19.412315 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2l5gl" event={"ID":"7d4917d4-c3ba-432c-83b2-0672de4a70f8","Type":"ContainerStarted","Data":"07cdcb67bb92dfaff50cdab623aa03baea4b6b2ef46b85b39714c5c70dafda0a"} Jul 07 15:40:19.877431 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:19.876592 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret\") pod \"global-pull-secret-syncer-q2m8b\" (UID: \"b99c24f7-0333-47fe-8b1a-ed9c6a10d23c\") " pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:19.877431 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:19.876645 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs\") pod \"network-metrics-daemon-tb5cw\" (UID: \"285707f9-4f1b-480e-ae8e-123d5ba608c6\") " pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:19.877431 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:19.876801 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:19.877431 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:19.876873 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs podName:285707f9-4f1b-480e-ae8e-123d5ba608c6 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:21.876851268 +0000 UTC m=+6.054495730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs") pod "network-metrics-daemon-tb5cw" (UID: "285707f9-4f1b-480e-ae8e-123d5ba608c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:19.877431 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:19.877332 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:19.877431 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:19.877388 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret podName:b99c24f7-0333-47fe-8b1a-ed9c6a10d23c nodeName:}" failed. No retries permitted until 2026-07-07 15:40:21.877372016 +0000 UTC m=+6.055016462 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret") pod "global-pull-secret-syncer-q2m8b" (UID: "b99c24f7-0333-47fe-8b1a-ed9c6a10d23c") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:20.078154 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:20.077971 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2klg\" (UniqueName: \"kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg\") pod \"network-check-target-hwwk7\" (UID: \"a767f075-0967-445e-8038-d22a7eda5b15\") " pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:20.078362 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:20.078208 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:40:20.078362 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:20.078234 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:40:20.078362 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:20.078247 2557 projected.go:196] Error preparing data for projected volume kube-api-access-m2klg for pod openshift-network-diagnostics/network-check-target-hwwk7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:20.078362 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:20.078307 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg podName:a767f075-0967-445e-8038-d22a7eda5b15 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:22.078288308 +0000 UTC m=+6.255932741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m2klg" (UniqueName: "kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg") pod "network-check-target-hwwk7" (UID: "a767f075-0967-445e-8038-d22a7eda5b15") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:20.289432 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:20.289293 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 15:35:18 +0000 UTC" deadline="2027-12-06 21:49:42.169788026 +0000 UTC" Jul 07 15:40:20.289432 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:20.289341 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12414h9m21.880451873s" Jul 07 15:40:20.356977 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:20.356945 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:20.357176 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:20.357092 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5cw" podUID="285707f9-4f1b-480e-ae8e-123d5ba608c6" Jul 07 15:40:20.357807 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:20.357769 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:20.357927 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:20.357886 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hwwk7" podUID="a767f075-0967-445e-8038-d22a7eda5b15" Jul 07 15:40:20.357988 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:20.357956 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:20.358085 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:20.358044 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2m8b" podUID="b99c24f7-0333-47fe-8b1a-ed9c6a10d23c" Jul 07 15:40:20.469653 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:20.469610 2557 generic.go:356] "Generic (PLEG): container finished" podID="f1f4c5f2ef966b7794d2c0682814a647" containerID="c1b115c706bf47d62044618f285282065d3089e620586929dfb95dcd35f31f1f" exitCode=0 Jul 07 15:40:20.469823 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:20.469680 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal" event={"ID":"f1f4c5f2ef966b7794d2c0682814a647","Type":"ContainerDied","Data":"c1b115c706bf47d62044618f285282065d3089e620586929dfb95dcd35f31f1f"} Jul 07 15:40:21.487666 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:21.487618 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal" event={"ID":"f1f4c5f2ef966b7794d2c0682814a647","Type":"ContainerStarted","Data":"bd27334029587452282eb7a2d62bcd71e787f3e1a4c766ba3c73f533b4d4479a"} Jul 07 15:40:21.508827 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:21.508761 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal" podStartSLOduration=4.5087132709999995 podStartE2EDuration="4.508713271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:40:21.508537896 +0000 UTC m=+5.686182351" watchObservedRunningTime="2026-07-07 15:40:21.508713271 +0000 UTC m=+5.686357727" Jul 07 15:40:21.893907 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:21.892942 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret\") pod \"global-pull-secret-syncer-q2m8b\" (UID: \"b99c24f7-0333-47fe-8b1a-ed9c6a10d23c\") " pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:21.893907 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:21.893002 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs\") pod \"network-metrics-daemon-tb5cw\" (UID: \"285707f9-4f1b-480e-ae8e-123d5ba608c6\") " pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:21.893907 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:21.893223 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:21.893907 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:21.893291 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs podName:285707f9-4f1b-480e-ae8e-123d5ba608c6 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:25.893271685 +0000 UTC m=+10.070916117 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs") pod "network-metrics-daemon-tb5cw" (UID: "285707f9-4f1b-480e-ae8e-123d5ba608c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:21.893907 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:21.893753 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:21.893907 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:21.893813 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret podName:b99c24f7-0333-47fe-8b1a-ed9c6a10d23c nodeName:}" failed. No retries permitted until 2026-07-07 15:40:25.893796588 +0000 UTC m=+10.071441021 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret") pod "global-pull-secret-syncer-q2m8b" (UID: "b99c24f7-0333-47fe-8b1a-ed9c6a10d23c") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:22.094535 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:22.094491 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2klg\" (UniqueName: \"kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg\") pod \"network-check-target-hwwk7\" (UID: \"a767f075-0967-445e-8038-d22a7eda5b15\") " pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:22.094774 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:22.094708 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:40:22.094774 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:22.094735 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:40:22.094774 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:22.094751 2557 projected.go:196] Error preparing data for projected volume kube-api-access-m2klg for pod openshift-network-diagnostics/network-check-target-hwwk7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:22.094977 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:22.094821 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg podName:a767f075-0967-445e-8038-d22a7eda5b15 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:26.094798621 +0000 UTC m=+10.272443055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m2klg" (UniqueName: "kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg") pod "network-check-target-hwwk7" (UID: "a767f075-0967-445e-8038-d22a7eda5b15") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:22.349708 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:22.348999 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:22.349708 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:22.349139 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hwwk7" podUID="a767f075-0967-445e-8038-d22a7eda5b15" Jul 07 15:40:22.349708 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:22.349336 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:22.349708 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:22.349454 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5cw" podUID="285707f9-4f1b-480e-ae8e-123d5ba608c6" Jul 07 15:40:22.349708 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:22.349521 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:22.349708 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:22.349599 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2m8b" podUID="b99c24f7-0333-47fe-8b1a-ed9c6a10d23c" Jul 07 15:40:24.345271 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:24.345212 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:24.345752 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:24.345215 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:24.345752 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:24.345216 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:24.345752 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:24.345436 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5cw" podUID="285707f9-4f1b-480e-ae8e-123d5ba608c6" Jul 07 15:40:24.345752 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:24.345533 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hwwk7" podUID="a767f075-0967-445e-8038-d22a7eda5b15" Jul 07 15:40:24.345752 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:24.345645 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2m8b" podUID="b99c24f7-0333-47fe-8b1a-ed9c6a10d23c" Jul 07 15:40:25.927295 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:25.927254 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret\") pod \"global-pull-secret-syncer-q2m8b\" (UID: \"b99c24f7-0333-47fe-8b1a-ed9c6a10d23c\") " pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:25.927758 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:25.927313 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs\") pod \"network-metrics-daemon-tb5cw\" (UID: \"285707f9-4f1b-480e-ae8e-123d5ba608c6\") " pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:25.927758 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:25.927475 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:25.927758 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:25.927538 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs podName:285707f9-4f1b-480e-ae8e-123d5ba608c6 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:33.92751996 +0000 UTC m=+18.105164403 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs") pod "network-metrics-daemon-tb5cw" (UID: "285707f9-4f1b-480e-ae8e-123d5ba608c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:25.928016 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:25.927984 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:25.928087 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:25.928041 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret podName:b99c24f7-0333-47fe-8b1a-ed9c6a10d23c nodeName:}" failed. No retries permitted until 2026-07-07 15:40:33.928025047 +0000 UTC m=+18.105669494 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret") pod "global-pull-secret-syncer-q2m8b" (UID: "b99c24f7-0333-47fe-8b1a-ed9c6a10d23c") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:26.128626 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:26.128580 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2klg\" (UniqueName: \"kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg\") pod \"network-check-target-hwwk7\" (UID: \"a767f075-0967-445e-8038-d22a7eda5b15\") " pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:26.128802 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:26.128767 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:40:26.128802 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:26.128791 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:40:26.128802 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:26.128801 2557 projected.go:196] Error preparing data for projected volume kube-api-access-m2klg for pod openshift-network-diagnostics/network-check-target-hwwk7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:26.128945 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:26.128863 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg podName:a767f075-0967-445e-8038-d22a7eda5b15 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:34.128844786 +0000 UTC m=+18.306489236 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m2klg" (UniqueName: "kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg") pod "network-check-target-hwwk7" (UID: "a767f075-0967-445e-8038-d22a7eda5b15") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:26.345654 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:26.345534 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:26.345654 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:26.345539 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:26.345878 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:26.345647 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2m8b" podUID="b99c24f7-0333-47fe-8b1a-ed9c6a10d23c" Jul 07 15:40:26.345878 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:26.345713 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hwwk7" podUID="a767f075-0967-445e-8038-d22a7eda5b15" Jul 07 15:40:26.345878 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:26.345543 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:26.345878 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:26.345817 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5cw" podUID="285707f9-4f1b-480e-ae8e-123d5ba608c6" Jul 07 15:40:28.345462 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:28.345373 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:28.345943 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:28.345505 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:28.345943 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:28.345521 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2m8b" podUID="b99c24f7-0333-47fe-8b1a-ed9c6a10d23c" Jul 07 15:40:28.345943 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:28.345614 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hwwk7" podUID="a767f075-0967-445e-8038-d22a7eda5b15" Jul 07 15:40:28.345943 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:28.345660 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:28.345943 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:28.345738 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5cw" podUID="285707f9-4f1b-480e-ae8e-123d5ba608c6" Jul 07 15:40:30.347577 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:30.347546 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:30.347972 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:30.347691 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:30.347972 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:30.347720 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:30.347972 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:30.347707 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2m8b" podUID="b99c24f7-0333-47fe-8b1a-ed9c6a10d23c" Jul 07 15:40:30.347972 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:30.347814 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5cw" podUID="285707f9-4f1b-480e-ae8e-123d5ba608c6" Jul 07 15:40:30.347972 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:30.347887 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hwwk7" podUID="a767f075-0967-445e-8038-d22a7eda5b15" Jul 07 15:40:32.344684 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:32.344638 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:32.344684 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:32.344657 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:32.344684 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:32.344638 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:32.345220 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:32.344792 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5cw" podUID="285707f9-4f1b-480e-ae8e-123d5ba608c6" Jul 07 15:40:32.345220 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:32.344863 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2m8b" podUID="b99c24f7-0333-47fe-8b1a-ed9c6a10d23c" Jul 07 15:40:32.345220 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:32.344881 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hwwk7" podUID="a767f075-0967-445e-8038-d22a7eda5b15" Jul 07 15:40:33.992220 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:33.992179 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret\") pod \"global-pull-secret-syncer-q2m8b\" (UID: \"b99c24f7-0333-47fe-8b1a-ed9c6a10d23c\") " pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:33.992762 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:33.992237 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs\") pod \"network-metrics-daemon-tb5cw\" (UID: \"285707f9-4f1b-480e-ae8e-123d5ba608c6\") " pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:33.992762 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:33.992351 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:33.992762 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:33.992380 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:33.992762 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:33.992419 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret podName:b99c24f7-0333-47fe-8b1a-ed9c6a10d23c nodeName:}" failed. No retries permitted until 2026-07-07 15:40:49.992403137 +0000 UTC m=+34.170047564 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret") pod "global-pull-secret-syncer-q2m8b" (UID: "b99c24f7-0333-47fe-8b1a-ed9c6a10d23c") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:33.992762 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:33.992435 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs podName:285707f9-4f1b-480e-ae8e-123d5ba608c6 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:49.992427415 +0000 UTC m=+34.170071844 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs") pod "network-metrics-daemon-tb5cw" (UID: "285707f9-4f1b-480e-ae8e-123d5ba608c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:34.194682 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:34.194638 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2klg\" (UniqueName: \"kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg\") pod \"network-check-target-hwwk7\" (UID: \"a767f075-0967-445e-8038-d22a7eda5b15\") " pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:34.194872 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:34.194850 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:40:34.194916 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:34.194880 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:40:34.194916 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:34.194896 2557 projected.go:196] Error preparing data for projected volume kube-api-access-m2klg for pod openshift-network-diagnostics/network-check-target-hwwk7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:34.194992 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:34.194964 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg podName:a767f075-0967-445e-8038-d22a7eda5b15 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:50.194943795 +0000 UTC m=+34.372588230 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m2klg" (UniqueName: "kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg") pod "network-check-target-hwwk7" (UID: "a767f075-0967-445e-8038-d22a7eda5b15") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:34.344682 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:34.344650 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:34.344865 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:34.344654 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:34.344865 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:34.344779 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2m8b" podUID="b99c24f7-0333-47fe-8b1a-ed9c6a10d23c" Jul 07 15:40:34.344865 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:34.344656 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:34.344865 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:34.344855 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5cw" podUID="285707f9-4f1b-480e-ae8e-123d5ba608c6" Jul 07 15:40:34.345067 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:34.344920 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hwwk7" podUID="a767f075-0967-445e-8038-d22a7eda5b15" Jul 07 15:40:36.345595 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:36.345559 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:36.346062 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:36.345671 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:36.346062 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:36.345681 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2m8b" podUID="b99c24f7-0333-47fe-8b1a-ed9c6a10d23c" Jul 07 15:40:36.346062 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:36.345698 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:36.346062 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:36.345806 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5cw" podUID="285707f9-4f1b-480e-ae8e-123d5ba608c6" Jul 07 15:40:36.346062 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:36.345896 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hwwk7" podUID="a767f075-0967-445e-8038-d22a7eda5b15" Jul 07 15:40:38.344845 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:38.344707 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:38.344845 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:38.344754 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:38.344845 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:38.344715 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:38.345405 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:38.344848 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2m8b" podUID="b99c24f7-0333-47fe-8b1a-ed9c6a10d23c" Jul 07 15:40:38.345405 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:38.344927 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5cw" podUID="285707f9-4f1b-480e-ae8e-123d5ba608c6" Jul 07 15:40:38.345405 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:38.345001 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hwwk7" podUID="a767f075-0967-445e-8038-d22a7eda5b15" Jul 07 15:40:39.525830 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:39.525623 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mcrlt" event={"ID":"78ecda82-54b3-4727-89b9-5d55dd0ca428","Type":"ContainerStarted","Data":"501aef68225d198cfaca3e80eeb7182886cae118db3e9a610dbf04f9fda1caa4"} Jul 07 15:40:39.527583 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:39.527376 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vktnc" event={"ID":"3567278c-a1c9-403d-9353-80da294fafb9","Type":"ContainerStarted","Data":"2483016f1ee5e82fdcaf31c8cacba518c3893ba7d2c773be4f07f3c7cf8eb3b8"} Jul 07 15:40:39.528841 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:39.528806 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lhvjn" event={"ID":"e7b29c32-fe48-437a-956a-f4b0d4eab6c3","Type":"ContainerStarted","Data":"4563f397270ad1169c3f1e1b503f57c73fa326a188d87e65004d139c30e122fd"} Jul 07 15:40:39.530844 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:39.530820 2557 generic.go:356] "Generic (PLEG): container finished" podID="7d4917d4-c3ba-432c-83b2-0672de4a70f8" containerID="4355c48227ca1f4e49f1f211700ce91576055d0a5bdb57b7f031e466684d0908" exitCode=0 Jul 07 15:40:39.530958 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:39.530902 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2l5gl" event={"ID":"7d4917d4-c3ba-432c-83b2-0672de4a70f8","Type":"ContainerDied","Data":"4355c48227ca1f4e49f1f211700ce91576055d0a5bdb57b7f031e466684d0908"} Jul 07 15:40:39.533355 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:39.533329 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f46rg" event={"ID":"78c8109c-5327-468b-ba1d-327ab10afd8e","Type":"ContainerStarted","Data":"e753e9c78d5191a6b53cae3a4334f5bbf537076b109cc282350bc61d4330af0f"} Jul 07 15:40:39.534980 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:39.534950 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" event={"ID":"acd15283-0678-4182-928a-cfd5f57c2923","Type":"ContainerStarted","Data":"0d327c3c250c5a009ccfb5cf1658df48c2071ed81c1e21d13ca4cde68df284da"} Jul 07 15:40:39.536908 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:39.536883 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" event={"ID":"c6660812-8311-4ffc-8966-bb7324e3afbb","Type":"ContainerStarted","Data":"60cc5dbcec742561e53d1e0e6d0160e6495b34eee3d80c72dc7dfe8d4c21e3c6"} Jul 07 15:40:39.538754 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:39.538727 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal" event={"ID":"d0a6e48ccbac6ae57118c8e5af923f7b","Type":"ContainerStarted","Data":"5f5ddc9f3216d9e92bfad3ea54d9bb22b58094fcd5e2d81b9f5705d7812c5061"} Jul 07 15:40:39.544121 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:39.542408 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hkj2z_07a80436-65b9-4f0d-99b3-ecda36240b48/ovn-acl-logging/0.log" Jul 07 15:40:39.546949 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:39.546151 2557 generic.go:356] "Generic (PLEG): container finished" podID="07a80436-65b9-4f0d-99b3-ecda36240b48" containerID="e74189077f24100bbdf5807a780c7327ebaaccbc77f6b5230f3eef018c7be117" exitCode=1 Jul 07 15:40:39.546949 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:39.546191 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" event={"ID":"07a80436-65b9-4f0d-99b3-ecda36240b48","Type":"ContainerStarted","Data":"37e2e5dbbd91218154af34a5731dc21e3e9f9fedb49c3723e78cfd823f804360"} Jul 07 15:40:39.546949 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:39.546423 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" event={"ID":"07a80436-65b9-4f0d-99b3-ecda36240b48","Type":"ContainerStarted","Data":"56e716c32214ef37a92e43b089106d19b299451f60b91268a0ac900754ac4cd4"} Jul 07 15:40:39.547203 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:39.544974 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mcrlt" podStartSLOduration=2.231077027 podStartE2EDuration="22.544955863s" totalImagesPullingTime="20.313878836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:40:39.543477288 +0000 UTC m=+23.721121723" watchObservedRunningTime="2026-07-07 15:40:39.544955863 +0000 UTC m=+23.722600312" Jul 07 15:40:39.547755 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:39.547435 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" event={"ID":"07a80436-65b9-4f0d-99b3-ecda36240b48","Type":"ContainerDied","Data":"e74189077f24100bbdf5807a780c7327ebaaccbc77f6b5230f3eef018c7be117"} Jul 07 15:40:39.547755 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:39.547517 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" event={"ID":"07a80436-65b9-4f0d-99b3-ecda36240b48","Type":"ContainerStarted","Data":"1e4b2a5acae689b19d272ead49dfd1cbc3684b494c14fe74bf5fcd40d44a9429"} Jul 07 15:40:39.564646 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:39.564603 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8r9n4" podStartSLOduration=3.21148376 podStartE2EDuration="23.564589248s" totalImagesPullingTime="20.353105488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:40:39.564222917 +0000 UTC m=+23.741867387" watchObservedRunningTime="2026-07-07 15:40:39.564589248 +0000 UTC m=+23.742233697" Jul 07 15:40:39.581544 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:39.581498 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-f46rg" podStartSLOduration=2.199366605 podStartE2EDuration="22.58148205s" totalImagesPullingTime="20.382115445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:40:39.58125593 +0000 UTC m=+23.758900379" watchObservedRunningTime="2026-07-07 15:40:39.58148205 +0000 UTC m=+23.759126500" Jul 07 15:40:39.626617 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:39.626550 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lhvjn" podStartSLOduration=2.321353418 podStartE2EDuration="22.626529652s" totalImagesPullingTime="20.305176234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:40:39.625994471 +0000 UTC m=+23.803638920" watchObservedRunningTime="2026-07-07 15:40:39.626529652 +0000 UTC m=+23.804174105" Jul 07 15:40:39.666814 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:39.666694 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vktnc" podStartSLOduration=2.392595855 podStartE2EDuration="22.666676159s" totalImagesPullingTime="20.274080304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:40:39.665643977 +0000 UTC m=+23.843288426" watchObservedRunningTime="2026-07-07 15:40:39.666676159 +0000 UTC m=+23.844320609" Jul 07 15:40:39.666814 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:39.666772 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal" podStartSLOduration=22.666768392 podStartE2EDuration="22.666768392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:40:39.642575399 +0000 UTC m=+23.820219851" watchObservedRunningTime="2026-07-07 15:40:39.666768392 +0000 UTC m=+23.844412878" Jul 07 15:40:40.345417 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:40.345332 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:40.345579 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:40.345332 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:40.345579 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:40.345469 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hwwk7" podUID="a767f075-0967-445e-8038-d22a7eda5b15" Jul 07 15:40:40.345579 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:40.345333 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:40.345579 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:40.345544 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5cw" podUID="285707f9-4f1b-480e-ae8e-123d5ba608c6" Jul 07 15:40:40.345789 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:40.345615 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2m8b" podUID="b99c24f7-0333-47fe-8b1a-ed9c6a10d23c" Jul 07 15:40:40.552697 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:40.552674 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hkj2z_07a80436-65b9-4f0d-99b3-ecda36240b48/ovn-acl-logging/0.log" Jul 07 15:40:40.553183 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:40.553092 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" event={"ID":"07a80436-65b9-4f0d-99b3-ecda36240b48","Type":"ContainerStarted","Data":"cd5007c8e0b42acf4123aa806b8b3609b451803e4f71bfa8cd7a434ba9b424bb"} Jul 07 15:40:40.553249 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:40.553199 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" event={"ID":"07a80436-65b9-4f0d-99b3-ecda36240b48","Type":"ContainerStarted","Data":"febb05d6c0186e47d1394be52fe317d53245b0fb7ec22f0c377822b02ada9dfa"} Jul 07 15:40:40.555251 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:40.555107 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q7l26" event={"ID":"0de85e66-128c-44a2-b566-5770f8b132e0","Type":"ContainerStarted","Data":"25650113e5a6ce6feb1a0a8f9c9ba83f5f2ea4f89291f07c13ed9733678ae8a4"} Jul 07 15:40:40.570055 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:40.570007 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-q7l26" podStartSLOduration=3.283561692 podStartE2EDuration="23.569986924s" totalImagesPullingTime="20.286425232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:40:40.569955007 +0000 UTC m=+24.747599478" watchObservedRunningTime="2026-07-07 15:40:40.569986924 +0000 UTC m=+24.747631375" Jul 07 15:40:40.612767 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:40.612743 2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 15:40:41.316297 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:41.315961 2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T15:40:40.612761605Z","UUID":"5d2b95ce-713e-438b-8da1-8c47fbb1561f","Handler":null,"Name":"","Endpoint":""} Jul 07 15:40:41.317886 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:41.317860 2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 15:40:41.318021 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:41.317908 2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 15:40:41.559256 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:41.559218 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" event={"ID":"c6660812-8311-4ffc-8966-bb7324e3afbb","Type":"ContainerStarted","Data":"e95d21cb50f26c08378d8ab5d1becf3d5e3111e14e902af77f6229f03300b87e"} Jul 07 15:40:42.344831 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:42.344796 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:42.345023 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:42.344795 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:42.345023 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:42.344934 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:42.345023 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:42.344934 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hwwk7" podUID="a767f075-0967-445e-8038-d22a7eda5b15" Jul 07 15:40:42.345231 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:42.345037 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5cw" podUID="285707f9-4f1b-480e-ae8e-123d5ba608c6" Jul 07 15:40:42.345231 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:42.345207 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2m8b" podUID="b99c24f7-0333-47fe-8b1a-ed9c6a10d23c" Jul 07 15:40:42.564317 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:42.564280 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" event={"ID":"c6660812-8311-4ffc-8966-bb7324e3afbb","Type":"ContainerStarted","Data":"0c8c9263ea8d97bfcd518dbd11ed0c3b7ed2f0720a331bc5dcc6a6f70634224a"} Jul 07 15:40:42.567906 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:42.567884 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hkj2z_07a80436-65b9-4f0d-99b3-ecda36240b48/ovn-acl-logging/0.log" Jul 07 15:40:42.568331 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:42.568288 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" event={"ID":"07a80436-65b9-4f0d-99b3-ecda36240b48","Type":"ContainerStarted","Data":"3cd56e2b5c6fcaaee2956eb51e7bb04963feddd655d7ae2f5fbf56da64bd4122"} Jul 07 15:40:42.582247 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:42.582204 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwfx6" podStartSLOduration=3.436252352 podStartE2EDuration="26.582190646s" totalImagesPullingTime="23.145938294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:40:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:40:42.581764783 +0000 UTC m=+26.759409254" watchObservedRunningTime="2026-07-07 15:40:42.582190646 +0000 UTC m=+26.759835095" Jul 07 15:40:44.335755 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:44.335708 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vktnc" Jul 07 15:40:44.336564 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:44.336545 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vktnc" Jul 07 15:40:44.344698 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:44.344672 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:44.344835 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:44.344677 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:44.344835 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:44.344814 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:44.344835 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:44.344812 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5cw" podUID="285707f9-4f1b-480e-ae8e-123d5ba608c6" Jul 07 15:40:44.345005 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:44.344919 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hwwk7" podUID="a767f075-0967-445e-8038-d22a7eda5b15" Jul 07 15:40:44.345072 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:44.345010 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2m8b" podUID="b99c24f7-0333-47fe-8b1a-ed9c6a10d23c" Jul 07 15:40:44.573438 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:44.573244 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vktnc" Jul 07 15:40:44.573969 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:44.573952 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vktnc" Jul 07 15:40:45.577090 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:45.577044 2557 generic.go:356] "Generic (PLEG): container finished" podID="7d4917d4-c3ba-432c-83b2-0672de4a70f8" containerID="4c43a29fe1782b92d8e1ccdc59f27d6bc12cadcff321ab378f36cd2381686acd" exitCode=0 Jul 07 15:40:45.577653 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:45.577159 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2l5gl" event={"ID":"7d4917d4-c3ba-432c-83b2-0672de4a70f8","Type":"ContainerDied","Data":"4c43a29fe1782b92d8e1ccdc59f27d6bc12cadcff321ab378f36cd2381686acd"} Jul 07 15:40:45.580711 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:45.580692 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hkj2z_07a80436-65b9-4f0d-99b3-ecda36240b48/ovn-acl-logging/0.log" Jul 07 15:40:45.581059 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:45.581022 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" event={"ID":"07a80436-65b9-4f0d-99b3-ecda36240b48","Type":"ContainerStarted","Data":"b20545fd8cf61c8a4f7a288f907dac934a6a8529737878433634dc7b94af806b"} Jul 07 15:40:45.581430 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:45.581410 2557 scope.go:117] "RemoveContainer" containerID="e74189077f24100bbdf5807a780c7327ebaaccbc77f6b5230f3eef018c7be117" Jul 07 15:40:45.581523 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:45.581430 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:45.581523 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:45.581487 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:45.581615 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:45.581526 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:45.597421 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:45.597396 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:45.597534 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:45.597499 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:40:46.345400 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:46.345363 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:46.345800 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:46.345473 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:46.345800 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:46.345480 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2m8b" podUID="b99c24f7-0333-47fe-8b1a-ed9c6a10d23c" Jul 07 15:40:46.345800 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:46.345590 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5cw" podUID="285707f9-4f1b-480e-ae8e-123d5ba608c6" Jul 07 15:40:46.345800 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:46.345648 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:46.345800 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:46.345726 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hwwk7" podUID="a767f075-0967-445e-8038-d22a7eda5b15" Jul 07 15:40:46.570229 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:46.570027 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q2m8b"] Jul 07 15:40:46.572621 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:46.572596 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tb5cw"] Jul 07 15:40:46.581939 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:46.581917 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hwwk7"] Jul 07 15:40:46.587249 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:46.587230 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hkj2z_07a80436-65b9-4f0d-99b3-ecda36240b48/ovn-acl-logging/0.log" Jul 07 15:40:46.587653 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:46.587635 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:46.587739 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:46.587683 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" event={"ID":"07a80436-65b9-4f0d-99b3-ecda36240b48","Type":"ContainerStarted","Data":"d9da33ccfc61c8afe73d9165f8788b6d316c31531e6b874f9282ec9664ef15d5"} Jul 07 15:40:46.587799 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:46.587743 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hwwk7" podUID="a767f075-0967-445e-8038-d22a7eda5b15" Jul 07 15:40:46.588037 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:46.588013 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:46.588191 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:46.588166 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2m8b" podUID="b99c24f7-0333-47fe-8b1a-ed9c6a10d23c" Jul 07 15:40:46.588191 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:46.588185 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:46.588321 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:46.588301 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5cw" podUID="285707f9-4f1b-480e-ae8e-123d5ba608c6" Jul 07 15:40:46.614795 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:46.614680 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" podStartSLOduration=9.262080382 podStartE2EDuration="29.614659438s" totalImagesPullingTime="20.352579056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:40:46.61416616 +0000 UTC m=+30.791810638" watchObservedRunningTime="2026-07-07 15:40:46.614659438 +0000 UTC m=+30.792303889" Jul 07 15:40:47.591461 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:47.591427 2557 generic.go:356] "Generic (PLEG): container finished" podID="7d4917d4-c3ba-432c-83b2-0672de4a70f8" containerID="361edcd2bba134968e3a8dc69ffeb5835e730b0b5a44a05a582fbe53f0cf63aa" exitCode=0 Jul 07 15:40:47.591864 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:47.591518 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2l5gl" event={"ID":"7d4917d4-c3ba-432c-83b2-0672de4a70f8","Type":"ContainerDied","Data":"361edcd2bba134968e3a8dc69ffeb5835e730b0b5a44a05a582fbe53f0cf63aa"} Jul 07 15:40:48.344580 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:48.344544 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:48.344580 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:48.344572 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:48.344893 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:48.344544 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:48.344893 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:48.344681 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5cw" podUID="285707f9-4f1b-480e-ae8e-123d5ba608c6" Jul 07 15:40:48.344893 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:48.344779 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2m8b" podUID="b99c24f7-0333-47fe-8b1a-ed9c6a10d23c" Jul 07 15:40:48.344893 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:48.344837 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hwwk7" podUID="a767f075-0967-445e-8038-d22a7eda5b15" Jul 07 15:40:49.597680 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:49.597648 2557 generic.go:356] "Generic (PLEG): container finished" podID="7d4917d4-c3ba-432c-83b2-0672de4a70f8" containerID="35f196e18f59ef3ec783dd8bd841a6c1480cc77401083d3497519b6c60b6bbc1" exitCode=0 Jul 07 15:40:49.598314 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:49.597707 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2l5gl" event={"ID":"7d4917d4-c3ba-432c-83b2-0672de4a70f8","Type":"ContainerDied","Data":"35f196e18f59ef3ec783dd8bd841a6c1480cc77401083d3497519b6c60b6bbc1"} Jul 07 15:40:50.005002 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:50.004902 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret\") pod \"global-pull-secret-syncer-q2m8b\" (UID: \"b99c24f7-0333-47fe-8b1a-ed9c6a10d23c\") " pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:50.005002 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:50.004944 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs\") pod \"network-metrics-daemon-tb5cw\" (UID: \"285707f9-4f1b-480e-ae8e-123d5ba608c6\") " pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:50.005228 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:50.005059 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:50.005228 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:50.005065 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:50.005228 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:50.005133 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs podName:285707f9-4f1b-480e-ae8e-123d5ba608c6 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:22.00509967 +0000 UTC m=+66.182744098 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs") pod "network-metrics-daemon-tb5cw" (UID: "285707f9-4f1b-480e-ae8e-123d5ba608c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:50.005228 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:50.005150 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret podName:b99c24f7-0333-47fe-8b1a-ed9c6a10d23c nodeName:}" failed. No retries permitted until 2026-07-07 15:41:22.005141127 +0000 UTC m=+66.182785555 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret") pod "global-pull-secret-syncer-q2m8b" (UID: "b99c24f7-0333-47fe-8b1a-ed9c6a10d23c") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:50.206524 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:50.206493 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2klg\" (UniqueName: \"kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg\") pod \"network-check-target-hwwk7\" (UID: \"a767f075-0967-445e-8038-d22a7eda5b15\") " pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:50.206700 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:50.206631 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:40:50.206700 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:50.206645 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:40:50.206700 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:50.206654 2557 projected.go:196] Error preparing data for projected volume kube-api-access-m2klg for pod openshift-network-diagnostics/network-check-target-hwwk7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:50.206793 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:50.206703 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg podName:a767f075-0967-445e-8038-d22a7eda5b15 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:22.206691414 +0000 UTC m=+66.384335842 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m2klg" (UniqueName: "kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg") pod "network-check-target-hwwk7" (UID: "a767f075-0967-445e-8038-d22a7eda5b15") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:50.345087 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:50.345052 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:50.345264 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:50.345103 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:50.345264 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:50.345052 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:50.345264 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:50.345196 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5cw" podUID="285707f9-4f1b-480e-ae8e-123d5ba608c6" Jul 07 15:40:50.345264 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:50.345258 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hwwk7" podUID="a767f075-0967-445e-8038-d22a7eda5b15" Jul 07 15:40:50.345410 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:50.345334 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2m8b" podUID="b99c24f7-0333-47fe-8b1a-ed9c6a10d23c" Jul 07 15:40:52.137619 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.137449 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeReady" Jul 07 15:40:52.137963 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.137710 2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 15:40:52.178032 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.177998 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-848545c49c-thmhm"] Jul 07 15:40:52.180292 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.180274 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.182466 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.182447 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 07 15:40:52.182822 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.182795 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 07 15:40:52.182935 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.182869 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kdgps\"" Jul 07 15:40:52.183008 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.182960 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 07 15:40:52.188346 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.188313 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 07 15:40:52.197224 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.197192 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-848545c49c-thmhm"] Jul 07 15:40:52.201892 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.201871 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8tdvw"] Jul 07 15:40:52.204274 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.204258 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8tdvw" Jul 07 15:40:52.206796 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.206684 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jrjtw"] Jul 07 15:40:52.208605 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.208513 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 07 15:40:52.208605 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.208563 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 07 15:40:52.208795 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.208731 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 07 15:40:52.209581 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.209089 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 07 15:40:52.211786 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.211689 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-l8qph\"" Jul 07 15:40:52.212782 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.212720 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-vsnfw"] Jul 07 15:40:52.213440 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.213334 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jrjtw" Jul 07 15:40:52.217881 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.217231 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7b5cfc467-zp2s8"] Jul 07 15:40:52.217881 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.217460 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 07 15:40:52.217881 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.217469 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-vsnfw" Jul 07 15:40:52.217881 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.217488 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 07 15:40:52.217881 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.217491 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 07 15:40:52.217881 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.217621 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-kk95r\"" Jul 07 15:40:52.219623 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.219604 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 07 15:40:52.220374 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.220354 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7z9gg\"" Jul 07 15:40:52.220481 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.220448 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 07 15:40:52.221005 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.220985 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-spwqp"] Jul 07 15:40:52.221132 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.221100 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:40:52.223193 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.223176 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-q6mbt\"" Jul 07 15:40:52.224455 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.224435 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j"] Jul 07 15:40:52.224739 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.224505 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-spwqp" Jul 07 15:40:52.225682 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.225417 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 07 15:40:52.225682 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.225426 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 07 15:40:52.226145 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.225919 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 15:40:52.226145 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.225951 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 07 15:40:52.226145 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.225977 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 07 15:40:52.227019 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.226634 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tn46g\"" Jul 07 15:40:52.227019 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.226859 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 15:40:52.227869 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.227850 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 15:40:52.230993 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.230642 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-v6x9c"] Jul 07 15:40:52.231259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.231239 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 15:40:52.231818 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.231802 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j" Jul 07 15:40:52.234129 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.234095 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 07 15:40:52.234757 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.234737 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4gzbc\"" Jul 07 15:40:52.234948 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.234900 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" Jul 07 15:40:52.235203 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.234819 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kt6w8"] Jul 07 15:40:52.235741 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.235701 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 15:40:52.236185 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.235741 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 15:40:52.236326 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.236310 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 07 15:40:52.239607 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.239587 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 07 15:40:52.239864 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.239592 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 15:40:52.240183 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.239633 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 15:40:52.241396 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.240885 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 07 15:40:52.241396 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.240970 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vt7qx\"" Jul 07 15:40:52.245611 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.245090 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-ss2mx"] Jul 07 15:40:52.246341 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.246319 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kt6w8" Jul 07 15:40:52.250591 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.250571 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 07 15:40:52.250940 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.250916 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rk7b5"] Jul 07 15:40:52.251185 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.251170 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ss2mx" Jul 07 15:40:52.252492 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.252474 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-ppm6m\"" Jul 07 15:40:52.253369 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.252510 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 07 15:40:52.253669 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.252693 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 07 15:40:52.256677 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.255627 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 15:40:52.256677 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.255869 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-kwscg\"" Jul 07 15:40:52.256677 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.252839 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 07 15:40:52.256677 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.252745 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 07 15:40:52.256677 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.256412 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 15:40:52.258959 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.258936 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-vfwk5"] Jul 07 15:40:52.259106 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.259091 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rk7b5" Jul 07 15:40:52.261082 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.261061 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 15:40:52.261226 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.261180 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 15:40:52.261521 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.261497 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 15:40:52.261737 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.261722 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tqcfj\"" Jul 07 15:40:52.262579 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.262462 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-dn2nk"] Jul 07 15:40:52.262955 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.262753 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-vfwk5" Jul 07 15:40:52.264325 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.264275 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 07 15:40:52.264418 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.264354 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9w2h6\"" Jul 07 15:40:52.264509 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.264490 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 07 15:40:52.265508 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.265488 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8tdvw"] Jul 07 15:40:52.265593 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.265526 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jrjtw"] Jul 07 15:40:52.265593 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.265554 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j"] Jul 07 15:40:52.265593 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.265574 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-v6x9c"] Jul 07 15:40:52.265593 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.265593 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7b5cfc467-zp2s8"] Jul 07 15:40:52.265810 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.265612 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-vsnfw"] Jul 07 15:40:52.265810 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.265751 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" Jul 07 15:40:52.265810 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.265782 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kt6w8"] Jul 07 15:40:52.265945 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.265815 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-dn2nk"] Jul 07 15:40:52.265945 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.265840 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-vfwk5"] Jul 07 15:40:52.265945 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.265863 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-spwqp"] Jul 07 15:40:52.265945 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.265886 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-ss2mx"] Jul 07 15:40:52.265945 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.265905 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rk7b5"] Jul 07 15:40:52.267859 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.267841 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 07 15:40:52.268069 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.268050 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 07 15:40:52.268230 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.268213 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 07 15:40:52.268358 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.268238 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 07 15:40:52.268358 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.268345 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-z2989\"" Jul 07 15:40:52.273408 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.273390 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 07 15:40:52.322744 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.322684 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/37e8dea9-9c9a-41f5-bd64-6d0331ced375-snapshots\") pod \"insights-operator-5c847dcfc4-v6x9c\" (UID: \"37e8dea9-9c9a-41f5-bd64-6d0331ced375\") " pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" Jul 07 15:40:52.322744 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.322723 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e8dea9-9c9a-41f5-bd64-6d0331ced375-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-v6x9c\" (UID: \"37e8dea9-9c9a-41f5-bd64-6d0331ced375\") " pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" Jul 07 15:40:52.323084 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.322754 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zgms\" (UniqueName: \"kubernetes.io/projected/37e8dea9-9c9a-41f5-bd64-6d0331ced375-kube-api-access-8zgms\") pod \"insights-operator-5c847dcfc4-v6x9c\" (UID: \"37e8dea9-9c9a-41f5-bd64-6d0331ced375\") " pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" Jul 07 15:40:52.323084 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.322779 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce7245b7-e2e6-43df-9ae2-568fb739ed49-installation-pull-secrets\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.323084 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.322807 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.323084 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.322837 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-default-certificate\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:40:52.323084 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.322861 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6937108d-00d9-456e-80a3-5230106a0708-service-ca-bundle\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:40:52.323084 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.322899 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-jrjtw\" (UID: \"8f635da8-3f7f-4450-bdc5-bb8616c2fb12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jrjtw" Jul 07 15:40:52.323084 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.322923 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-8mf6j\" (UID: \"cdd40d24-e30b-496c-b92b-7c80e6ee0b33\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j" Jul 07 15:40:52.323084 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.322946 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-tmp-dir\") pod \"dns-default-spwqp\" (UID: \"4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3\") " pod="openshift-dns/dns-default-spwqp" Jul 07 15:40:52.323084 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.322970 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pnst\" (UniqueName: \"kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-kube-api-access-4pnst\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.323084 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.322998 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq6g9\" (UniqueName: \"kubernetes.io/projected/acc35711-b1b5-4807-9228-70b97021ffb6-kube-api-access-nq6g9\") pod \"service-ca-operator-5d46d9ffd4-8tdvw\" (UID: \"acc35711-b1b5-4807-9228-70b97021ffb6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8tdvw" Jul 07 15:40:52.323084 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323022 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-stats-auth\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:40:52.323084 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323060 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e8dea9-9c9a-41f5-bd64-6d0331ced375-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-v6x9c\" (UID: \"37e8dea9-9c9a-41f5-bd64-6d0331ced375\") " pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" Jul 07 15:40:52.323084 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323077 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce7245b7-e2e6-43df-9ae2-568fb739ed49-trusted-ca\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.323608 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323093 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-bound-sa-token\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.323608 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323128 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-metrics-certs\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:40:52.323608 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323172 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37e8dea9-9c9a-41f5-bd64-6d0331ced375-tmp\") pod \"insights-operator-5c847dcfc4-v6x9c\" (UID: \"37e8dea9-9c9a-41f5-bd64-6d0331ced375\") " pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" Jul 07 15:40:52.323608 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323198 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37e8dea9-9c9a-41f5-bd64-6d0331ced375-serving-cert\") pod \"insights-operator-5c847dcfc4-v6x9c\" (UID: \"37e8dea9-9c9a-41f5-bd64-6d0331ced375\") " pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" Jul 07 15:40:52.323608 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323247 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acc35711-b1b5-4807-9228-70b97021ffb6-config\") pod \"service-ca-operator-5d46d9ffd4-8tdvw\" (UID: \"acc35711-b1b5-4807-9228-70b97021ffb6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8tdvw" Jul 07 15:40:52.323608 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323275 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-metrics-tls\") pod \"dns-default-spwqp\" (UID: \"4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3\") " pod="openshift-dns/dns-default-spwqp" Jul 07 15:40:52.323608 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323306 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62417042-0998-4d6a-ba6a-71ec72bde8e6-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-kt6w8\" (UID: \"62417042-0998-4d6a-ba6a-71ec72bde8e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kt6w8" Jul 07 15:40:52.323608 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323362 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4tw5\" (UniqueName: \"kubernetes.io/projected/55083ac2-3c7e-4664-995d-4d0ab74e3a20-kube-api-access-v4tw5\") pod \"volume-data-source-validator-5d456fdbcd-vsnfw\" (UID: \"55083ac2-3c7e-4664-995d-4d0ab74e3a20\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-vsnfw" Jul 07 15:40:52.323608 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323386 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-certificates\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.323608 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323406 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms24h\" (UniqueName: \"kubernetes.io/projected/62417042-0998-4d6a-ba6a-71ec72bde8e6-kube-api-access-ms24h\") pod \"kube-storage-version-migrator-operator-846bf668f7-kt6w8\" (UID: \"62417042-0998-4d6a-ba6a-71ec72bde8e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kt6w8" Jul 07 15:40:52.323608 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323436 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbrpf\" (UniqueName: \"kubernetes.io/projected/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-kube-api-access-tbrpf\") pod \"cluster-samples-operator-56fcf86bf-jrjtw\" (UID: \"8f635da8-3f7f-4450-bdc5-bb8616c2fb12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jrjtw" Jul 07 15:40:52.323608 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323457 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-config-volume\") pod \"dns-default-spwqp\" (UID: \"4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3\") " pod="openshift-dns/dns-default-spwqp" Jul 07 15:40:52.323608 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323484 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/acc35711-b1b5-4807-9228-70b97021ffb6-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-8tdvw\" (UID: \"acc35711-b1b5-4807-9228-70b97021ffb6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8tdvw" Jul 07 15:40:52.323608 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323506 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds6p8\" (UniqueName: \"kubernetes.io/projected/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-kube-api-access-ds6p8\") pod \"cluster-monitoring-operator-64786d9cbb-8mf6j\" (UID: \"cdd40d24-e30b-496c-b92b-7c80e6ee0b33\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j" Jul 07 15:40:52.323608 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323533 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8mf6j\" (UID: \"cdd40d24-e30b-496c-b92b-7c80e6ee0b33\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j" Jul 07 15:40:52.324164 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323557 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p2wd\" (UniqueName: \"kubernetes.io/projected/6937108d-00d9-456e-80a3-5230106a0708-kube-api-access-4p2wd\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:40:52.324164 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323602 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qv5z\" (UniqueName: \"kubernetes.io/projected/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-kube-api-access-9qv5z\") pod \"dns-default-spwqp\" (UID: \"4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3\") " pod="openshift-dns/dns-default-spwqp" Jul 07 15:40:52.324164 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323715 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62417042-0998-4d6a-ba6a-71ec72bde8e6-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-kt6w8\" (UID: \"62417042-0998-4d6a-ba6a-71ec72bde8e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kt6w8" Jul 07 15:40:52.324164 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323769 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce7245b7-e2e6-43df-9ae2-568fb739ed49-ca-trust-extracted\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.324164 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.323823 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce7245b7-e2e6-43df-9ae2-568fb739ed49-image-registry-private-configuration\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.343223 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.343186 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:40:52.345122 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.345099 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:40:52.345180 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.345132 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:40:52.345220 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.345103 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:40:52.347334 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.347306 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 15:40:52.347334 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.347319 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jh4tr\"" Jul 07 15:40:52.347597 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.347579 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v8g69\"" Jul 07 15:40:52.347654 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.347596 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 15:40:52.424354 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.424317 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-default-certificate\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:40:52.424543 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.424366 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6937108d-00d9-456e-80a3-5230106a0708-service-ca-bundle\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:40:52.424543 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.424400 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-jrjtw\" (UID: \"8f635da8-3f7f-4450-bdc5-bb8616c2fb12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jrjtw" Jul 07 15:40:52.424543 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.424488 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 15:40:52.424707 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.424559 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-samples-operator-tls podName:8f635da8-3f7f-4450-bdc5-bb8616c2fb12 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:52.924538173 +0000 UTC m=+37.102182603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-jrjtw" (UID: "8f635da8-3f7f-4450-bdc5-bb8616c2fb12") : secret "samples-operator-tls" not found Jul 07 15:40:52.424707 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.424596 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-8mf6j\" (UID: \"cdd40d24-e30b-496c-b92b-7c80e6ee0b33\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j" Jul 07 15:40:52.424707 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.424649 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-tmp-dir\") pod \"dns-default-spwqp\" (UID: \"4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3\") " pod="openshift-dns/dns-default-spwqp" Jul 07 15:40:52.424707 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.424677 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pnst\" (UniqueName: \"kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-kube-api-access-4pnst\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.424925 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.424727 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq6g9\" (UniqueName: \"kubernetes.io/projected/acc35711-b1b5-4807-9228-70b97021ffb6-kube-api-access-nq6g9\") pod \"service-ca-operator-5d46d9ffd4-8tdvw\" (UID: \"acc35711-b1b5-4807-9228-70b97021ffb6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8tdvw" Jul 07 15:40:52.424925 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.424759 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-stats-auth\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:40:52.424925 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.424789 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a278ee42-38b2-4aee-b359-602e3bbf59be-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-vfwk5\" (UID: \"a278ee42-38b2-4aee-b359-602e3bbf59be\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-vfwk5" Jul 07 15:40:52.424925 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.424819 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e8dea9-9c9a-41f5-bd64-6d0331ced375-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-v6x9c\" (UID: \"37e8dea9-9c9a-41f5-bd64-6d0331ced375\") " pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" Jul 07 15:40:52.424925 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.424843 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce7245b7-e2e6-43df-9ae2-568fb739ed49-trusted-ca\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.424925 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.424865 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-bound-sa-token\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.424925 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.424889 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-metrics-certs\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:40:52.424925 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.424915 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37e8dea9-9c9a-41f5-bd64-6d0331ced375-tmp\") pod \"insights-operator-5c847dcfc4-v6x9c\" (UID: \"37e8dea9-9c9a-41f5-bd64-6d0331ced375\") " pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" Jul 07 15:40:52.425332 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.424941 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37e8dea9-9c9a-41f5-bd64-6d0331ced375-serving-cert\") pod \"insights-operator-5c847dcfc4-v6x9c\" (UID: \"37e8dea9-9c9a-41f5-bd64-6d0331ced375\") " pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" Jul 07 15:40:52.425332 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.424965 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acc35711-b1b5-4807-9228-70b97021ffb6-config\") pod \"service-ca-operator-5d46d9ffd4-8tdvw\" (UID: \"acc35711-b1b5-4807-9228-70b97021ffb6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8tdvw" Jul 07 15:40:52.425332 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.424988 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-metrics-tls\") pod \"dns-default-spwqp\" (UID: \"4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3\") " pod="openshift-dns/dns-default-spwqp" Jul 07 15:40:52.425332 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.425011 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62417042-0998-4d6a-ba6a-71ec72bde8e6-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-kt6w8\" (UID: \"62417042-0998-4d6a-ba6a-71ec72bde8e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kt6w8" Jul 07 15:40:52.425332 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.425231 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8stql\" (UniqueName: \"kubernetes.io/projected/0f8f2e78-29ca-4b2a-b9d1-fb08f60d5fbc-kube-api-access-8stql\") pod \"network-check-source-989f8b8dc-ss2mx\" (UID: \"0f8f2e78-29ca-4b2a-b9d1-fb08f60d5fbc\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ss2mx" Jul 07 15:40:52.425332 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.425268 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4tw5\" (UniqueName: \"kubernetes.io/projected/55083ac2-3c7e-4664-995d-4d0ab74e3a20-kube-api-access-v4tw5\") pod \"volume-data-source-validator-5d456fdbcd-vsnfw\" (UID: \"55083ac2-3c7e-4664-995d-4d0ab74e3a20\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-vsnfw" Jul 07 15:40:52.425332 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.425299 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-certificates\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.425332 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.425325 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c9vt\" (UniqueName: \"kubernetes.io/projected/d1fdbe6c-661b-4201-87f1-7ca6fe2b561b-kube-api-access-2c9vt\") pod \"console-operator-68b757fb67-dn2nk\" (UID: \"d1fdbe6c-661b-4201-87f1-7ca6fe2b561b\") " pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" Jul 07 15:40:52.425699 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.425661 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms24h\" (UniqueName: \"kubernetes.io/projected/62417042-0998-4d6a-ba6a-71ec72bde8e6-kube-api-access-ms24h\") pod \"kube-storage-version-migrator-operator-846bf668f7-kt6w8\" (UID: \"62417042-0998-4d6a-ba6a-71ec72bde8e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kt6w8" Jul 07 15:40:52.425854 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.425831 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37e8dea9-9c9a-41f5-bd64-6d0331ced375-tmp\") pod \"insights-operator-5c847dcfc4-v6x9c\" (UID: \"37e8dea9-9c9a-41f5-bd64-6d0331ced375\") " pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" Jul 07 15:40:52.425926 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.425909 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 15:40:52.425974 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.425950 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-8mf6j\" (UID: \"cdd40d24-e30b-496c-b92b-7c80e6ee0b33\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j" Jul 07 15:40:52.425974 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.425949 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e8dea9-9c9a-41f5-bd64-6d0331ced375-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-v6x9c\" (UID: \"37e8dea9-9c9a-41f5-bd64-6d0331ced375\") " pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" Jul 07 15:40:52.426089 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.425978 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-metrics-tls podName:4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:52.925961341 +0000 UTC m=+37.103605782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-metrics-tls") pod "dns-default-spwqp" (UID: "4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3") : secret "dns-default-metrics-tls" not found Jul 07 15:40:52.427240 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.426288 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 15:40:52.427240 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.426356 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-metrics-certs podName:6937108d-00d9-456e-80a3-5230106a0708 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:52.926339937 +0000 UTC m=+37.103984386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-metrics-certs") pod "router-default-7b5cfc467-zp2s8" (UID: "6937108d-00d9-456e-80a3-5230106a0708") : secret "router-metrics-certs-default" not found Jul 07 15:40:52.427240 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.426424 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbrpf\" (UniqueName: \"kubernetes.io/projected/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-kube-api-access-tbrpf\") pod \"cluster-samples-operator-56fcf86bf-jrjtw\" (UID: \"8f635da8-3f7f-4450-bdc5-bb8616c2fb12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jrjtw" Jul 07 15:40:52.427240 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.426452 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-config-volume\") pod \"dns-default-spwqp\" (UID: \"4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3\") " pod="openshift-dns/dns-default-spwqp" Jul 07 15:40:52.427240 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.426483 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/acc35711-b1b5-4807-9228-70b97021ffb6-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-8tdvw\" (UID: \"acc35711-b1b5-4807-9228-70b97021ffb6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8tdvw" Jul 07 15:40:52.427240 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.426511 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds6p8\" (UniqueName: \"kubernetes.io/projected/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-kube-api-access-ds6p8\") pod \"cluster-monitoring-operator-64786d9cbb-8mf6j\" (UID: \"cdd40d24-e30b-496c-b92b-7c80e6ee0b33\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j" Jul 07 15:40:52.427240 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.426539 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xb9n\" (UniqueName: \"kubernetes.io/projected/a9aad135-443d-43c6-8b9d-94107c57ef21-kube-api-access-5xb9n\") pod \"ingress-canary-rk7b5\" (UID: \"a9aad135-443d-43c6-8b9d-94107c57ef21\") " pod="openshift-ingress-canary/ingress-canary-rk7b5" Jul 07 15:40:52.427240 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.426575 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8mf6j\" (UID: \"cdd40d24-e30b-496c-b92b-7c80e6ee0b33\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j" Jul 07 15:40:52.427240 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.426606 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p2wd\" (UniqueName: \"kubernetes.io/projected/6937108d-00d9-456e-80a3-5230106a0708-kube-api-access-4p2wd\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:40:52.427240 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.426634 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qv5z\" (UniqueName: \"kubernetes.io/projected/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-kube-api-access-9qv5z\") pod \"dns-default-spwqp\" (UID: \"4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3\") " pod="openshift-dns/dns-default-spwqp" Jul 07 15:40:52.427240 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.426659 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d1fdbe6c-661b-4201-87f1-7ca6fe2b561b-config\") pod \"console-operator-68b757fb67-dn2nk\" (UID: \"d1fdbe6c-661b-4201-87f1-7ca6fe2b561b\") " pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" Jul 07 15:40:52.427240 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.426682 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1fdbe6c-661b-4201-87f1-7ca6fe2b561b-serving-cert\") pod \"console-operator-68b757fb67-dn2nk\" (UID: \"d1fdbe6c-661b-4201-87f1-7ca6fe2b561b\") " pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" Jul 07 15:40:52.427240 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.426706 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-certificates\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.427240 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.426709 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9aad135-443d-43c6-8b9d-94107c57ef21-cert\") pod \"ingress-canary-rk7b5\" (UID: \"a9aad135-443d-43c6-8b9d-94107c57ef21\") " pod="openshift-ingress-canary/ingress-canary-rk7b5" Jul 07 15:40:52.427240 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.426767 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62417042-0998-4d6a-ba6a-71ec72bde8e6-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-kt6w8\" (UID: \"62417042-0998-4d6a-ba6a-71ec72bde8e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kt6w8" Jul 07 15:40:52.427240 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.426789 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 15:40:52.428047 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.426796 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce7245b7-e2e6-43df-9ae2-568fb739ed49-ca-trust-extracted\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.428047 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.426841 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1fdbe6c-661b-4201-87f1-7ca6fe2b561b-trusted-ca\") pod \"console-operator-68b757fb67-dn2nk\" (UID: \"d1fdbe6c-661b-4201-87f1-7ca6fe2b561b\") " pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" Jul 07 15:40:52.428047 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.426897 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce7245b7-e2e6-43df-9ae2-568fb739ed49-image-registry-private-configuration\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.428047 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.426930 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/37e8dea9-9c9a-41f5-bd64-6d0331ced375-snapshots\") pod \"insights-operator-5c847dcfc4-v6x9c\" (UID: \"37e8dea9-9c9a-41f5-bd64-6d0331ced375\") " pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" Jul 07 15:40:52.428047 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.426959 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e8dea9-9c9a-41f5-bd64-6d0331ced375-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-v6x9c\" (UID: \"37e8dea9-9c9a-41f5-bd64-6d0331ced375\") " pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" Jul 07 15:40:52.428047 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.426966 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce7245b7-e2e6-43df-9ae2-568fb739ed49-trusted-ca\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.428047 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.427004 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zgms\" (UniqueName: \"kubernetes.io/projected/37e8dea9-9c9a-41f5-bd64-6d0331ced375-kube-api-access-8zgms\") pod \"insights-operator-5c847dcfc4-v6x9c\" (UID: \"37e8dea9-9c9a-41f5-bd64-6d0331ced375\") " pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" Jul 07 15:40:52.428047 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.426976 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-config-volume\") pod \"dns-default-spwqp\" (UID: \"4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3\") " pod="openshift-dns/dns-default-spwqp" Jul 07 15:40:52.428047 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.427039 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-cluster-monitoring-operator-tls podName:cdd40d24-e30b-496c-b92b-7c80e6ee0b33 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:52.927020541 +0000 UTC m=+37.104664969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8mf6j" (UID: "cdd40d24-e30b-496c-b92b-7c80e6ee0b33") : secret "cluster-monitoring-operator-tls" not found Jul 07 15:40:52.428047 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.427062 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce7245b7-e2e6-43df-9ae2-568fb739ed49-installation-pull-secrets\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.428047 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.427091 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a278ee42-38b2-4aee-b359-602e3bbf59be-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-vfwk5\" (UID: \"a278ee42-38b2-4aee-b359-602e3bbf59be\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-vfwk5" Jul 07 15:40:52.428047 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.427143 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.428047 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.427269 2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 15:40:52.428047 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.427283 2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-848545c49c-thmhm: secret "image-registry-tls" not found Jul 07 15:40:52.428047 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.427304 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce7245b7-e2e6-43df-9ae2-568fb739ed49-ca-trust-extracted\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.428789 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.427322 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls podName:ce7245b7-e2e6-43df-9ae2-568fb739ed49 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:52.927309135 +0000 UTC m=+37.104953568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls") pod "image-registry-848545c49c-thmhm" (UID: "ce7245b7-e2e6-43df-9ae2-568fb739ed49") : secret "image-registry-tls" not found Jul 07 15:40:52.428789 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.427460 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6937108d-00d9-456e-80a3-5230106a0708-service-ca-bundle podName:6937108d-00d9-456e-80a3-5230106a0708 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:52.927441832 +0000 UTC m=+37.105086283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6937108d-00d9-456e-80a3-5230106a0708-service-ca-bundle") pod "router-default-7b5cfc467-zp2s8" (UID: "6937108d-00d9-456e-80a3-5230106a0708") : configmap references non-existent config key: service-ca.crt Jul 07 15:40:52.428789 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.428031 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/37e8dea9-9c9a-41f5-bd64-6d0331ced375-snapshots\") pod \"insights-operator-5c847dcfc4-v6x9c\" (UID: \"37e8dea9-9c9a-41f5-bd64-6d0331ced375\") " pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" Jul 07 15:40:52.428789 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.428565 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e8dea9-9c9a-41f5-bd64-6d0331ced375-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-v6x9c\" (UID: \"37e8dea9-9c9a-41f5-bd64-6d0331ced375\") " pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" Jul 07 15:40:52.428789 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.428690 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-tmp-dir\") pod \"dns-default-spwqp\" (UID: \"4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3\") " pod="openshift-dns/dns-default-spwqp" Jul 07 15:40:52.429054 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.428788 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acc35711-b1b5-4807-9228-70b97021ffb6-config\") pod \"service-ca-operator-5d46d9ffd4-8tdvw\" (UID: \"acc35711-b1b5-4807-9228-70b97021ffb6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8tdvw" Jul 07 15:40:52.429549 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.429528 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62417042-0998-4d6a-ba6a-71ec72bde8e6-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-kt6w8\" (UID: \"62417042-0998-4d6a-ba6a-71ec72bde8e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kt6w8" Jul 07 15:40:52.430324 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.430297 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/acc35711-b1b5-4807-9228-70b97021ffb6-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-8tdvw\" (UID: \"acc35711-b1b5-4807-9228-70b97021ffb6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8tdvw" Jul 07 15:40:52.430780 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.430717 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-default-certificate\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:40:52.430959 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.430870 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-stats-auth\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:40:52.431090 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.431064 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37e8dea9-9c9a-41f5-bd64-6d0331ced375-serving-cert\") pod \"insights-operator-5c847dcfc4-v6x9c\" (UID: \"37e8dea9-9c9a-41f5-bd64-6d0331ced375\") " pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" Jul 07 15:40:52.431532 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.431512 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce7245b7-e2e6-43df-9ae2-568fb739ed49-installation-pull-secrets\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.431948 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.431925 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62417042-0998-4d6a-ba6a-71ec72bde8e6-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-kt6w8\" (UID: \"62417042-0998-4d6a-ba6a-71ec72bde8e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kt6w8" Jul 07 15:40:52.432091 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.432071 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce7245b7-e2e6-43df-9ae2-568fb739ed49-image-registry-private-configuration\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.435755 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.435704 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds6p8\" (UniqueName: \"kubernetes.io/projected/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-kube-api-access-ds6p8\") pod \"cluster-monitoring-operator-64786d9cbb-8mf6j\" (UID: \"cdd40d24-e30b-496c-b92b-7c80e6ee0b33\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j" Jul 07 15:40:52.436056 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.435913 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:40:52.436286 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.436260 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-bound-sa-token\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.437708 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.437684 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qv5z\" (UniqueName: \"kubernetes.io/projected/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-kube-api-access-9qv5z\") pod \"dns-default-spwqp\" (UID: \"4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3\") " pod="openshift-dns/dns-default-spwqp" Jul 07 15:40:52.437902 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.437881 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq6g9\" (UniqueName: \"kubernetes.io/projected/acc35711-b1b5-4807-9228-70b97021ffb6-kube-api-access-nq6g9\") pod \"service-ca-operator-5d46d9ffd4-8tdvw\" (UID: \"acc35711-b1b5-4807-9228-70b97021ffb6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8tdvw" Jul 07 15:40:52.438410 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.438386 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms24h\" (UniqueName: \"kubernetes.io/projected/62417042-0998-4d6a-ba6a-71ec72bde8e6-kube-api-access-ms24h\") pod \"kube-storage-version-migrator-operator-846bf668f7-kt6w8\" (UID: \"62417042-0998-4d6a-ba6a-71ec72bde8e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kt6w8" Jul 07 15:40:52.438618 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.438598 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p2wd\" (UniqueName: \"kubernetes.io/projected/6937108d-00d9-456e-80a3-5230106a0708-kube-api-access-4p2wd\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:40:52.439091 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.439071 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zgms\" (UniqueName: \"kubernetes.io/projected/37e8dea9-9c9a-41f5-bd64-6d0331ced375-kube-api-access-8zgms\") pod \"insights-operator-5c847dcfc4-v6x9c\" (UID: \"37e8dea9-9c9a-41f5-bd64-6d0331ced375\") " pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" Jul 07 15:40:52.439204 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.439090 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4tw5\" (UniqueName: \"kubernetes.io/projected/55083ac2-3c7e-4664-995d-4d0ab74e3a20-kube-api-access-v4tw5\") pod \"volume-data-source-validator-5d456fdbcd-vsnfw\" (UID: \"55083ac2-3c7e-4664-995d-4d0ab74e3a20\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-vsnfw" Jul 07 15:40:52.440559 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.440483 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbrpf\" (UniqueName: \"kubernetes.io/projected/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-kube-api-access-tbrpf\") pod \"cluster-samples-operator-56fcf86bf-jrjtw\" (UID: \"8f635da8-3f7f-4450-bdc5-bb8616c2fb12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jrjtw" Jul 07 15:40:52.440559 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.440519 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pnst\" (UniqueName: \"kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-kube-api-access-4pnst\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.520619 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.520544 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8tdvw" Jul 07 15:40:52.527816 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.527783 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a278ee42-38b2-4aee-b359-602e3bbf59be-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-vfwk5\" (UID: \"a278ee42-38b2-4aee-b359-602e3bbf59be\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-vfwk5" Jul 07 15:40:52.527969 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.527869 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8stql\" (UniqueName: \"kubernetes.io/projected/0f8f2e78-29ca-4b2a-b9d1-fb08f60d5fbc-kube-api-access-8stql\") pod \"network-check-source-989f8b8dc-ss2mx\" (UID: \"0f8f2e78-29ca-4b2a-b9d1-fb08f60d5fbc\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ss2mx" Jul 07 15:40:52.527969 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.527901 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c9vt\" (UniqueName: \"kubernetes.io/projected/d1fdbe6c-661b-4201-87f1-7ca6fe2b561b-kube-api-access-2c9vt\") pod \"console-operator-68b757fb67-dn2nk\" (UID: \"d1fdbe6c-661b-4201-87f1-7ca6fe2b561b\") " pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" Jul 07 15:40:52.527969 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.527946 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xb9n\" (UniqueName: \"kubernetes.io/projected/a9aad135-443d-43c6-8b9d-94107c57ef21-kube-api-access-5xb9n\") pod \"ingress-canary-rk7b5\" (UID: \"a9aad135-443d-43c6-8b9d-94107c57ef21\") " pod="openshift-ingress-canary/ingress-canary-rk7b5" Jul 07 15:40:52.528134 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.527979 2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 15:40:52.528134 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.528002 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d1fdbe6c-661b-4201-87f1-7ca6fe2b561b-config\") pod \"console-operator-68b757fb67-dn2nk\" (UID: \"d1fdbe6c-661b-4201-87f1-7ca6fe2b561b\") " pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" Jul 07 15:40:52.528134 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.528026 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1fdbe6c-661b-4201-87f1-7ca6fe2b561b-serving-cert\") pod \"console-operator-68b757fb67-dn2nk\" (UID: \"d1fdbe6c-661b-4201-87f1-7ca6fe2b561b\") " pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" Jul 07 15:40:52.528134 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.528059 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a278ee42-38b2-4aee-b359-602e3bbf59be-networking-console-plugin-cert podName:a278ee42-38b2-4aee-b359-602e3bbf59be nodeName:}" failed. No retries permitted until 2026-07-07 15:40:53.028036799 +0000 UTC m=+37.205681241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a278ee42-38b2-4aee-b359-602e3bbf59be-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-vfwk5" (UID: "a278ee42-38b2-4aee-b359-602e3bbf59be") : secret "networking-console-plugin-cert" not found Jul 07 15:40:52.528134 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.528099 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9aad135-443d-43c6-8b9d-94107c57ef21-cert\") pod \"ingress-canary-rk7b5\" (UID: \"a9aad135-443d-43c6-8b9d-94107c57ef21\") " pod="openshift-ingress-canary/ingress-canary-rk7b5" Jul 07 15:40:52.528433 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.528172 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1fdbe6c-661b-4201-87f1-7ca6fe2b561b-trusted-ca\") pod \"console-operator-68b757fb67-dn2nk\" (UID: \"d1fdbe6c-661b-4201-87f1-7ca6fe2b561b\") " pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" Jul 07 15:40:52.528433 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.528223 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a278ee42-38b2-4aee-b359-602e3bbf59be-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-vfwk5\" (UID: \"a278ee42-38b2-4aee-b359-602e3bbf59be\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-vfwk5" Jul 07 15:40:52.528987 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.528963 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a278ee42-38b2-4aee-b359-602e3bbf59be-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-vfwk5\" (UID: \"a278ee42-38b2-4aee-b359-602e3bbf59be\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-vfwk5" Jul 07 15:40:52.529097 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.529054 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d1fdbe6c-661b-4201-87f1-7ca6fe2b561b-config\") pod \"console-operator-68b757fb67-dn2nk\" (UID: \"d1fdbe6c-661b-4201-87f1-7ca6fe2b561b\") " pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" Jul 07 15:40:52.529379 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.529355 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1fdbe6c-661b-4201-87f1-7ca6fe2b561b-trusted-ca\") pod \"console-operator-68b757fb67-dn2nk\" (UID: \"d1fdbe6c-661b-4201-87f1-7ca6fe2b561b\") " pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" Jul 07 15:40:52.531271 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.531249 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1fdbe6c-661b-4201-87f1-7ca6fe2b561b-serving-cert\") pod \"console-operator-68b757fb67-dn2nk\" (UID: \"d1fdbe6c-661b-4201-87f1-7ca6fe2b561b\") " pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" Jul 07 15:40:52.531378 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.531353 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9aad135-443d-43c6-8b9d-94107c57ef21-cert\") pod \"ingress-canary-rk7b5\" (UID: \"a9aad135-443d-43c6-8b9d-94107c57ef21\") " pod="openshift-ingress-canary/ingress-canary-rk7b5" Jul 07 15:40:52.536933 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.536906 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c9vt\" (UniqueName: \"kubernetes.io/projected/d1fdbe6c-661b-4201-87f1-7ca6fe2b561b-kube-api-access-2c9vt\") pod \"console-operator-68b757fb67-dn2nk\" (UID: \"d1fdbe6c-661b-4201-87f1-7ca6fe2b561b\") " pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" Jul 07 15:40:52.537058 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.537027 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xb9n\" (UniqueName: \"kubernetes.io/projected/a9aad135-443d-43c6-8b9d-94107c57ef21-kube-api-access-5xb9n\") pod \"ingress-canary-rk7b5\" (UID: \"a9aad135-443d-43c6-8b9d-94107c57ef21\") " pod="openshift-ingress-canary/ingress-canary-rk7b5" Jul 07 15:40:52.537150 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.537098 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8stql\" (UniqueName: \"kubernetes.io/projected/0f8f2e78-29ca-4b2a-b9d1-fb08f60d5fbc-kube-api-access-8stql\") pod \"network-check-source-989f8b8dc-ss2mx\" (UID: \"0f8f2e78-29ca-4b2a-b9d1-fb08f60d5fbc\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ss2mx" Jul 07 15:40:52.543731 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.543709 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-vsnfw" Jul 07 15:40:52.571863 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.571722 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kt6w8" Jul 07 15:40:52.593002 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.592887 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" Jul 07 15:40:52.599029 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.598950 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ss2mx" Jul 07 15:40:52.608532 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.608323 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rk7b5" Jul 07 15:40:52.619469 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.619261 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" Jul 07 15:40:52.731199 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.731070 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8tdvw"] Jul 07 15:40:52.737471 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.732977 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-vsnfw"] Jul 07 15:40:52.753334 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:52.753297 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc35711_b1b5_4807_9228_70b97021ffb6.slice/crio-c413614c23f720cacd7968791830a83ade31d8692392b64cdaea439fe59c71f2 WatchSource:0}: Error finding container c413614c23f720cacd7968791830a83ade31d8692392b64cdaea439fe59c71f2: Status 404 returned error can't find the container with id c413614c23f720cacd7968791830a83ade31d8692392b64cdaea439fe59c71f2 Jul 07 15:40:52.800905 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.800871 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kt6w8"] Jul 07 15:40:52.805590 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:52.805488 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62417042_0998_4d6a_ba6a_71ec72bde8e6.slice/crio-fb7c65be12fd47c146b5724555ee9ab34eac86d483cc1958b51a26e0be8e042c WatchSource:0}: Error finding container fb7c65be12fd47c146b5724555ee9ab34eac86d483cc1958b51a26e0be8e042c: Status 404 returned error can't find the container with id fb7c65be12fd47c146b5724555ee9ab34eac86d483cc1958b51a26e0be8e042c Jul 07 15:40:52.840226 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.840177 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-ss2mx"] Jul 07 15:40:52.843455 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:52.843426 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8f2e78_29ca_4b2a_b9d1_fb08f60d5fbc.slice/crio-6e444854c4dc1c6465da86bd395e8a38eea6dafb2a9422c3faa7b48edcc57fc8 WatchSource:0}: Error finding container 6e444854c4dc1c6465da86bd395e8a38eea6dafb2a9422c3faa7b48edcc57fc8: Status 404 returned error can't find the container with id 6e444854c4dc1c6465da86bd395e8a38eea6dafb2a9422c3faa7b48edcc57fc8 Jul 07 15:40:52.847673 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.847574 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-v6x9c"] Jul 07 15:40:52.851258 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:52.851234 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e8dea9_9c9a_41f5_bd64_6d0331ced375.slice/crio-2643d9895cddffdbe31e7ebaa8ba8a8252abd06d4075163c4f7f6b7938e952f8 WatchSource:0}: Error finding container 2643d9895cddffdbe31e7ebaa8ba8a8252abd06d4075163c4f7f6b7938e952f8: Status 404 returned error can't find the container with id 2643d9895cddffdbe31e7ebaa8ba8a8252abd06d4075163c4f7f6b7938e952f8 Jul 07 15:40:52.855439 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.855415 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-dn2nk"] Jul 07 15:40:52.858918 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:52.858885 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1fdbe6c_661b_4201_87f1_7ca6fe2b561b.slice/crio-1852583b6bb5ad6dd054b6b46470a177c565333bde52a0bb3ff0c02181ad7d2a WatchSource:0}: Error finding container 1852583b6bb5ad6dd054b6b46470a177c565333bde52a0bb3ff0c02181ad7d2a: Status 404 returned error can't find the container with id 1852583b6bb5ad6dd054b6b46470a177c565333bde52a0bb3ff0c02181ad7d2a Jul 07 15:40:52.859246 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.859227 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rk7b5"] Jul 07 15:40:52.861969 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:40:52.861944 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9aad135_443d_43c6_8b9d_94107c57ef21.slice/crio-b9c07c289b28b52f4fd4d5a2832f8ffa9ee58a3427790f3c64e1275848528e75 WatchSource:0}: Error finding container b9c07c289b28b52f4fd4d5a2832f8ffa9ee58a3427790f3c64e1275848528e75: Status 404 returned error can't find the container with id b9c07c289b28b52f4fd4d5a2832f8ffa9ee58a3427790f3c64e1275848528e75 Jul 07 15:40:52.932497 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.932399 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:52.932497 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.932452 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6937108d-00d9-456e-80a3-5230106a0708-service-ca-bundle\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:40:52.932727 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.932558 2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 15:40:52.932727 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.932579 2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-848545c49c-thmhm: secret "image-registry-tls" not found Jul 07 15:40:52.932727 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.932646 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6937108d-00d9-456e-80a3-5230106a0708-service-ca-bundle podName:6937108d-00d9-456e-80a3-5230106a0708 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:53.932617149 +0000 UTC m=+38.110261580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6937108d-00d9-456e-80a3-5230106a0708-service-ca-bundle") pod "router-default-7b5cfc467-zp2s8" (UID: "6937108d-00d9-456e-80a3-5230106a0708") : configmap references non-existent config key: service-ca.crt Jul 07 15:40:52.932727 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.932671 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls podName:ce7245b7-e2e6-43df-9ae2-568fb739ed49 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:53.932661486 +0000 UTC m=+38.110305917 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls") pod "image-registry-848545c49c-thmhm" (UID: "ce7245b7-e2e6-43df-9ae2-568fb739ed49") : secret "image-registry-tls" not found Jul 07 15:40:52.932727 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.932692 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-jrjtw\" (UID: \"8f635da8-3f7f-4450-bdc5-bb8616c2fb12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jrjtw" Jul 07 15:40:52.933055 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.932756 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-metrics-certs\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:40:52.933055 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.932794 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-metrics-tls\") pod \"dns-default-spwqp\" (UID: \"4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3\") " pod="openshift-dns/dns-default-spwqp" Jul 07 15:40:52.933055 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.932814 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 15:40:52.933055 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.932857 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-samples-operator-tls podName:8f635da8-3f7f-4450-bdc5-bb8616c2fb12 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:53.93284127 +0000 UTC m=+38.110485717 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-jrjtw" (UID: "8f635da8-3f7f-4450-bdc5-bb8616c2fb12") : secret "samples-operator-tls" not found Jul 07 15:40:52.933055 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.932868 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 15:40:52.933055 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.932899 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 15:40:52.933055 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:52.932905 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8mf6j\" (UID: \"cdd40d24-e30b-496c-b92b-7c80e6ee0b33\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j" Jul 07 15:40:52.933055 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.932918 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-metrics-certs podName:6937108d-00d9-456e-80a3-5230106a0708 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:53.932903643 +0000 UTC m=+38.110548073 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-metrics-certs") pod "router-default-7b5cfc467-zp2s8" (UID: "6937108d-00d9-456e-80a3-5230106a0708") : secret "router-metrics-certs-default" not found Jul 07 15:40:52.933055 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.932945 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-metrics-tls podName:4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:53.932934063 +0000 UTC m=+38.110578493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-metrics-tls") pod "dns-default-spwqp" (UID: "4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3") : secret "dns-default-metrics-tls" not found Jul 07 15:40:52.933055 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.932962 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 15:40:52.933055 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:52.932995 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-cluster-monitoring-operator-tls podName:cdd40d24-e30b-496c-b92b-7c80e6ee0b33 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:53.932983604 +0000 UTC m=+38.110628031 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8mf6j" (UID: "cdd40d24-e30b-496c-b92b-7c80e6ee0b33") : secret "cluster-monitoring-operator-tls" not found Jul 07 15:40:53.034611 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:53.034565 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a278ee42-38b2-4aee-b359-602e3bbf59be-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-vfwk5\" (UID: \"a278ee42-38b2-4aee-b359-602e3bbf59be\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-vfwk5" Jul 07 15:40:53.034783 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:53.034743 2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 15:40:53.034865 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:53.034829 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a278ee42-38b2-4aee-b359-602e3bbf59be-networking-console-plugin-cert podName:a278ee42-38b2-4aee-b359-602e3bbf59be nodeName:}" failed. No retries permitted until 2026-07-07 15:40:54.034803836 +0000 UTC m=+38.212448264 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a278ee42-38b2-4aee-b359-602e3bbf59be-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-vfwk5" (UID: "a278ee42-38b2-4aee-b359-602e3bbf59be") : secret "networking-console-plugin-cert" not found Jul 07 15:40:53.620036 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:53.619897 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rk7b5" event={"ID":"a9aad135-443d-43c6-8b9d-94107c57ef21","Type":"ContainerStarted","Data":"b9c07c289b28b52f4fd4d5a2832f8ffa9ee58a3427790f3c64e1275848528e75"} Jul 07 15:40:53.624580 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:53.624547 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-vsnfw" event={"ID":"55083ac2-3c7e-4664-995d-4d0ab74e3a20","Type":"ContainerStarted","Data":"7bc5d0e82a75704c1d3663775db7fbc593ebcf16e43f2062a867154554043de1"} Jul 07 15:40:53.628969 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:53.628896 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8tdvw" event={"ID":"acc35711-b1b5-4807-9228-70b97021ffb6","Type":"ContainerStarted","Data":"c413614c23f720cacd7968791830a83ade31d8692392b64cdaea439fe59c71f2"} Jul 07 15:40:53.632747 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:53.632720 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" event={"ID":"d1fdbe6c-661b-4201-87f1-7ca6fe2b561b","Type":"ContainerStarted","Data":"1852583b6bb5ad6dd054b6b46470a177c565333bde52a0bb3ff0c02181ad7d2a"} Jul 07 15:40:53.637322 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:53.637288 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kt6w8" event={"ID":"62417042-0998-4d6a-ba6a-71ec72bde8e6","Type":"ContainerStarted","Data":"fb7c65be12fd47c146b5724555ee9ab34eac86d483cc1958b51a26e0be8e042c"} Jul 07 15:40:53.639972 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:53.639912 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ss2mx" event={"ID":"0f8f2e78-29ca-4b2a-b9d1-fb08f60d5fbc","Type":"ContainerStarted","Data":"6e444854c4dc1c6465da86bd395e8a38eea6dafb2a9422c3faa7b48edcc57fc8"} Jul 07 15:40:53.646885 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:53.646858 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" event={"ID":"37e8dea9-9c9a-41f5-bd64-6d0331ced375","Type":"ContainerStarted","Data":"2643d9895cddffdbe31e7ebaa8ba8a8252abd06d4075163c4f7f6b7938e952f8"} Jul 07 15:40:53.945666 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:53.945575 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-metrics-certs\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:40:53.945666 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:53.945637 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-metrics-tls\") pod \"dns-default-spwqp\" (UID: \"4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3\") " pod="openshift-dns/dns-default-spwqp" Jul 07 15:40:53.945875 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:53.945709 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8mf6j\" (UID: \"cdd40d24-e30b-496c-b92b-7c80e6ee0b33\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j" Jul 07 15:40:53.945875 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:53.945788 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:53.945875 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:53.945820 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6937108d-00d9-456e-80a3-5230106a0708-service-ca-bundle\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:40:53.945875 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:53.945851 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-jrjtw\" (UID: \"8f635da8-3f7f-4450-bdc5-bb8616c2fb12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jrjtw" Jul 07 15:40:53.946065 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:53.945984 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 15:40:53.946065 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:53.946047 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-samples-operator-tls podName:8f635da8-3f7f-4450-bdc5-bb8616c2fb12 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:55.946029446 +0000 UTC m=+40.123673878 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-jrjtw" (UID: "8f635da8-3f7f-4450-bdc5-bb8616c2fb12") : secret "samples-operator-tls" not found Jul 07 15:40:53.946538 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:53.946518 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 15:40:53.946608 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:53.946576 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-metrics-certs podName:6937108d-00d9-456e-80a3-5230106a0708 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:55.946560988 +0000 UTC m=+40.124205421 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-metrics-certs") pod "router-default-7b5cfc467-zp2s8" (UID: "6937108d-00d9-456e-80a3-5230106a0708") : secret "router-metrics-certs-default" not found Jul 07 15:40:53.946667 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:53.946634 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 15:40:53.946667 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:53.946666 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-metrics-tls podName:4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:55.946654315 +0000 UTC m=+40.124298746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-metrics-tls") pod "dns-default-spwqp" (UID: "4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3") : secret "dns-default-metrics-tls" not found Jul 07 15:40:53.946771 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:53.946716 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 15:40:53.946771 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:53.946745 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-cluster-monitoring-operator-tls podName:cdd40d24-e30b-496c-b92b-7c80e6ee0b33 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:55.946735686 +0000 UTC m=+40.124380116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8mf6j" (UID: "cdd40d24-e30b-496c-b92b-7c80e6ee0b33") : secret "cluster-monitoring-operator-tls" not found Jul 07 15:40:53.946876 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:53.946803 2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 15:40:53.946876 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:53.946813 2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-848545c49c-thmhm: secret "image-registry-tls" not found Jul 07 15:40:53.946876 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:53.946842 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls podName:ce7245b7-e2e6-43df-9ae2-568fb739ed49 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:55.946832122 +0000 UTC m=+40.124476567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls") pod "image-registry-848545c49c-thmhm" (UID: "ce7245b7-e2e6-43df-9ae2-568fb739ed49") : secret "image-registry-tls" not found Jul 07 15:40:53.947016 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:53.946900 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6937108d-00d9-456e-80a3-5230106a0708-service-ca-bundle podName:6937108d-00d9-456e-80a3-5230106a0708 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:55.946890222 +0000 UTC m=+40.124534653 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6937108d-00d9-456e-80a3-5230106a0708-service-ca-bundle") pod "router-default-7b5cfc467-zp2s8" (UID: "6937108d-00d9-456e-80a3-5230106a0708") : configmap references non-existent config key: service-ca.crt Jul 07 15:40:54.047135 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:54.047083 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a278ee42-38b2-4aee-b359-602e3bbf59be-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-vfwk5\" (UID: \"a278ee42-38b2-4aee-b359-602e3bbf59be\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-vfwk5" Jul 07 15:40:54.047320 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:54.047224 2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 15:40:54.047320 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:54.047311 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a278ee42-38b2-4aee-b359-602e3bbf59be-networking-console-plugin-cert podName:a278ee42-38b2-4aee-b359-602e3bbf59be nodeName:}" failed. No retries permitted until 2026-07-07 15:40:56.047291745 +0000 UTC m=+40.224936195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a278ee42-38b2-4aee-b359-602e3bbf59be-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-vfwk5" (UID: "a278ee42-38b2-4aee-b359-602e3bbf59be") : secret "networking-console-plugin-cert" not found Jul 07 15:40:54.102054 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:54.102018 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:40:54.172416 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:54.172362 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:40:55.125625 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:55.125377 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:40:55.185653 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:55.185558 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:40:55.968497 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:55.968460 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6937108d-00d9-456e-80a3-5230106a0708-service-ca-bundle\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:40:55.968687 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:55.968507 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-jrjtw\" (UID: \"8f635da8-3f7f-4450-bdc5-bb8616c2fb12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jrjtw" Jul 07 15:40:55.968687 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:55.968561 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-metrics-certs\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:40:55.968687 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:55.968594 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-metrics-tls\") pod \"dns-default-spwqp\" (UID: \"4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3\") " pod="openshift-dns/dns-default-spwqp" Jul 07 15:40:55.968687 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:55.968664 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6937108d-00d9-456e-80a3-5230106a0708-service-ca-bundle podName:6937108d-00d9-456e-80a3-5230106a0708 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:59.968639407 +0000 UTC m=+44.146283851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6937108d-00d9-456e-80a3-5230106a0708-service-ca-bundle") pod "router-default-7b5cfc467-zp2s8" (UID: "6937108d-00d9-456e-80a3-5230106a0708") : configmap references non-existent config key: service-ca.crt Jul 07 15:40:55.968912 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:55.968702 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 15:40:55.968912 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:55.968708 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 15:40:55.968912 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:55.968721 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8mf6j\" (UID: \"cdd40d24-e30b-496c-b92b-7c80e6ee0b33\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j" Jul 07 15:40:55.968912 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:55.968756 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-metrics-certs podName:6937108d-00d9-456e-80a3-5230106a0708 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:59.968745129 +0000 UTC m=+44.146389562 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-metrics-certs") pod "router-default-7b5cfc467-zp2s8" (UID: "6937108d-00d9-456e-80a3-5230106a0708") : secret "router-metrics-certs-default" not found Jul 07 15:40:55.968912 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:55.968774 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-metrics-tls podName:4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:59.96876489 +0000 UTC m=+44.146409331 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-metrics-tls") pod "dns-default-spwqp" (UID: "4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3") : secret "dns-default-metrics-tls" not found Jul 07 15:40:55.968912 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:55.968793 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 15:40:55.968912 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:55.968810 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 15:40:55.968912 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:55.968851 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-cluster-monitoring-operator-tls podName:cdd40d24-e30b-496c-b92b-7c80e6ee0b33 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:59.968834984 +0000 UTC m=+44.146479417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8mf6j" (UID: "cdd40d24-e30b-496c-b92b-7c80e6ee0b33") : secret "cluster-monitoring-operator-tls" not found Jul 07 15:40:55.968912 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:55.968872 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:40:55.969302 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:55.968922 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-samples-operator-tls podName:8f635da8-3f7f-4450-bdc5-bb8616c2fb12 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:59.968912496 +0000 UTC m=+44.146556924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-jrjtw" (UID: "8f635da8-3f7f-4450-bdc5-bb8616c2fb12") : secret "samples-operator-tls" not found Jul 07 15:40:55.969302 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:55.968923 2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 15:40:55.969302 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:55.968939 2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-848545c49c-thmhm: secret "image-registry-tls" not found Jul 07 15:40:55.969302 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:55.968980 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls podName:ce7245b7-e2e6-43df-9ae2-568fb739ed49 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:59.968964476 +0000 UTC m=+44.146608925 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls") pod "image-registry-848545c49c-thmhm" (UID: "ce7245b7-e2e6-43df-9ae2-568fb739ed49") : secret "image-registry-tls" not found Jul 07 15:40:56.070344 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:56.070300 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a278ee42-38b2-4aee-b359-602e3bbf59be-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-vfwk5\" (UID: \"a278ee42-38b2-4aee-b359-602e3bbf59be\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-vfwk5" Jul 07 15:40:56.070522 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:56.070457 2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 15:40:56.070589 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:40:56.070543 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a278ee42-38b2-4aee-b359-602e3bbf59be-networking-console-plugin-cert podName:a278ee42-38b2-4aee-b359-602e3bbf59be nodeName:}" failed. No retries permitted until 2026-07-07 15:41:00.070521294 +0000 UTC m=+44.248165739 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a278ee42-38b2-4aee-b359-602e3bbf59be-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-vfwk5" (UID: "a278ee42-38b2-4aee-b359-602e3bbf59be") : secret "networking-console-plugin-cert" not found Jul 07 15:40:56.583345 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:56.583306 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:40:56.638263 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:56.638225 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:40:57.062936 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:57.062895 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:40:59.486806 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:59.486773 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:40:59.546695 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:40:59.546652 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:00.003417 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:00.003376 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8mf6j\" (UID: \"cdd40d24-e30b-496c-b92b-7c80e6ee0b33\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j" Jul 07 15:41:00.003600 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:00.003479 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:41:00.003600 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:00.003517 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6937108d-00d9-456e-80a3-5230106a0708-service-ca-bundle\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:41:00.003600 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:00.003540 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 15:41:00.003757 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:00.003622 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-cluster-monitoring-operator-tls podName:cdd40d24-e30b-496c-b92b-7c80e6ee0b33 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:08.003593179 +0000 UTC m=+52.181237607 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8mf6j" (UID: "cdd40d24-e30b-496c-b92b-7c80e6ee0b33") : secret "cluster-monitoring-operator-tls" not found Jul 07 15:41:00.003757 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:00.003642 2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 15:41:00.003757 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:00.003547 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-jrjtw\" (UID: \"8f635da8-3f7f-4450-bdc5-bb8616c2fb12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jrjtw" Jul 07 15:41:00.003757 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:00.003662 2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-848545c49c-thmhm: secret "image-registry-tls" not found Jul 07 15:41:00.003757 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:00.003644 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 15:41:00.003757 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:00.003700 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6937108d-00d9-456e-80a3-5230106a0708-service-ca-bundle podName:6937108d-00d9-456e-80a3-5230106a0708 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:08.003678448 +0000 UTC m=+52.181322894 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6937108d-00d9-456e-80a3-5230106a0708-service-ca-bundle") pod "router-default-7b5cfc467-zp2s8" (UID: "6937108d-00d9-456e-80a3-5230106a0708") : configmap references non-existent config key: service-ca.crt Jul 07 15:41:00.003757 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:00.003729 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls podName:ce7245b7-e2e6-43df-9ae2-568fb739ed49 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:08.003716431 +0000 UTC m=+52.181360861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls") pod "image-registry-848545c49c-thmhm" (UID: "ce7245b7-e2e6-43df-9ae2-568fb739ed49") : secret "image-registry-tls" not found Jul 07 15:41:00.004141 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:00.003784 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-metrics-certs\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:41:00.004141 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:00.003844 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-samples-operator-tls podName:8f635da8-3f7f-4450-bdc5-bb8616c2fb12 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:08.003821671 +0000 UTC m=+52.181466114 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-jrjtw" (UID: "8f635da8-3f7f-4450-bdc5-bb8616c2fb12") : secret "samples-operator-tls" not found Jul 07 15:41:00.004141 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:00.003875 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-metrics-tls\") pod \"dns-default-spwqp\" (UID: \"4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3\") " pod="openshift-dns/dns-default-spwqp" Jul 07 15:41:00.004141 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:00.003886 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 15:41:00.004141 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:00.003926 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-metrics-certs podName:6937108d-00d9-456e-80a3-5230106a0708 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:08.003914537 +0000 UTC m=+52.181558978 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-metrics-certs") pod "router-default-7b5cfc467-zp2s8" (UID: "6937108d-00d9-456e-80a3-5230106a0708") : secret "router-metrics-certs-default" not found Jul 07 15:41:00.004141 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:00.004027 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 15:41:00.004141 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:00.004098 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-metrics-tls podName:4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:08.004084734 +0000 UTC m=+52.181729171 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-metrics-tls") pod "dns-default-spwqp" (UID: "4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3") : secret "dns-default-metrics-tls" not found Jul 07 15:41:00.105343 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:00.105240 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a278ee42-38b2-4aee-b359-602e3bbf59be-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-vfwk5\" (UID: \"a278ee42-38b2-4aee-b359-602e3bbf59be\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-vfwk5" Jul 07 15:41:00.105526 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:00.105423 2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 15:41:00.105526 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:00.105508 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a278ee42-38b2-4aee-b359-602e3bbf59be-networking-console-plugin-cert podName:a278ee42-38b2-4aee-b359-602e3bbf59be nodeName:}" failed. No retries permitted until 2026-07-07 15:41:08.105485113 +0000 UTC m=+52.283129553 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a278ee42-38b2-4aee-b359-602e3bbf59be-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-vfwk5" (UID: "a278ee42-38b2-4aee-b359-602e3bbf59be") : secret "networking-console-plugin-cert" not found Jul 07 15:41:02.069555 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:02.069477 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:02.174929 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:02.174903 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:02.676710 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:02.676670 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ss2mx" event={"ID":"0f8f2e78-29ca-4b2a-b9d1-fb08f60d5fbc","Type":"ContainerStarted","Data":"41808fea414f469ae7c36fdc2fce9a3c06466afde17f3258d533ace1a8d91854"} Jul 07 15:41:02.679145 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:02.678966 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" event={"ID":"37e8dea9-9c9a-41f5-bd64-6d0331ced375","Type":"ContainerStarted","Data":"9b4af3ed0b3ddcc95f318762013974eb6f05ae46a4d599fa23e7b7f49110f8e1"} Jul 07 15:41:02.680967 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:02.680910 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rk7b5" event={"ID":"a9aad135-443d-43c6-8b9d-94107c57ef21","Type":"ContainerStarted","Data":"64e848a485c774c600b8ea6d99a87eecae53f008b8e3365e9d57437a1e03d7f9"} Jul 07 15:41:02.684280 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:02.684254 2557 generic.go:356] "Generic (PLEG): container finished" podID="7d4917d4-c3ba-432c-83b2-0672de4a70f8" containerID="7c88cab0131069d979ab9536fa19025b8f06c06448911d2d47c23435ca7ea597" exitCode=0 Jul 07 15:41:02.684412 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:02.684372 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2l5gl" event={"ID":"7d4917d4-c3ba-432c-83b2-0672de4a70f8","Type":"ContainerDied","Data":"7c88cab0131069d979ab9536fa19025b8f06c06448911d2d47c23435ca7ea597"} Jul 07 15:41:02.686293 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:02.686269 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-vsnfw" event={"ID":"55083ac2-3c7e-4664-995d-4d0ab74e3a20","Type":"ContainerStarted","Data":"2c00d421e29d454e3b29966a75fc084938cfb15eb6a2a751acf45465c9542fa6"} Jul 07 15:41:02.688477 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:02.688434 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8tdvw" event={"ID":"acc35711-b1b5-4807-9228-70b97021ffb6","Type":"ContainerStarted","Data":"c50f53bc41ce004f746c6ee69d8a820546dcc410ff87bd48a1a829b4949b362b"} Jul 07 15:41:02.690523 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:02.690506 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/0.log" Jul 07 15:41:02.690663 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:02.690648 2557 generic.go:356] "Generic (PLEG): container finished" podID="d1fdbe6c-661b-4201-87f1-7ca6fe2b561b" containerID="790cd4da8542369aaedcb0ac10c280a5a0a6611acda1069fb066af60069b0c95" exitCode=255 Jul 07 15:41:02.691069 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:02.690763 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" event={"ID":"d1fdbe6c-661b-4201-87f1-7ca6fe2b561b","Type":"ContainerDied","Data":"790cd4da8542369aaedcb0ac10c280a5a0a6611acda1069fb066af60069b0c95"} Jul 07 15:41:02.691230 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:02.690989 2557 scope.go:117] "RemoveContainer" containerID="790cd4da8542369aaedcb0ac10c280a5a0a6611acda1069fb066af60069b0c95" Jul 07 15:41:02.694490 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:02.694445 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ss2mx" podStartSLOduration=31.818582263 podStartE2EDuration="40.694426061s" totalImagesPullingTime="8.875843798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:02.693251542 +0000 UTC m=+46.870895994" watchObservedRunningTime="2026-07-07 15:41:02.694426061 +0000 UTC m=+46.872070512" Jul 07 15:41:02.694667 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:02.694646 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kt6w8" event={"ID":"62417042-0998-4d6a-ba6a-71ec72bde8e6","Type":"ContainerStarted","Data":"1c51fb596023aebaf20d442b0b0bde9900a2380b02ec9df7798acc05e0312d9b"} Jul 07 15:41:02.714086 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:02.713729 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rk7b5" podStartSLOduration=1.658735043 podStartE2EDuration="10.713708434s" totalImagesPullingTime="9.054973391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:02.713166874 +0000 UTC m=+46.890811352" watchObservedRunningTime="2026-07-07 15:41:02.713708434 +0000 UTC m=+46.891352878" Jul 07 15:41:02.730550 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:02.730072 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-vsnfw" podStartSLOduration=33.831228215 podStartE2EDuration="42.730052498s" totalImagesPullingTime="8.898824283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:02.728696424 +0000 UTC m=+46.906340875" watchObservedRunningTime="2026-07-07 15:41:02.730052498 +0000 UTC m=+46.907696952" Jul 07 15:41:02.748942 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:02.747913 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" podStartSLOduration=33.880208684 podStartE2EDuration="42.747892232s" totalImagesPullingTime="8.867683548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:02.746882585 +0000 UTC m=+46.924527059" watchObservedRunningTime="2026-07-07 15:41:02.747892232 +0000 UTC m=+46.925536687" Jul 07 15:41:02.822044 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:02.820655 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:02.822234 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:02.822060 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8tdvw" podStartSLOduration=33.849806268 podStartE2EDuration="42.822040605s" totalImagesPullingTime="8.972234337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:02.821630498 +0000 UTC m=+46.999274949" watchObservedRunningTime="2026-07-07 15:41:02.822040605 +0000 UTC m=+46.999685057" Jul 07 15:41:02.840969 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:02.840415 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kt6w8" podStartSLOduration=33.92702668 podStartE2EDuration="42.840389156s" totalImagesPullingTime="8.913362476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:02.839368761 +0000 UTC m=+47.017013217" watchObservedRunningTime="2026-07-07 15:41:02.840389156 +0000 UTC m=+47.018033602" Jul 07 15:41:03.128047 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.128006 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:03.185243 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.185208 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:03.367479 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.367408 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:03.472593 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.472557 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:03.481489 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.481445 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-q4x4h"] Jul 07 15:41:03.505271 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.505239 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q4x4h"] Jul 07 15:41:03.505423 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.505360 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:03.507383 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.507357 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 15:41:03.507514 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.507387 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fwwjf\"" Jul 07 15:41:03.507623 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.507608 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 15:41:03.538651 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.538627 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0eb10328-3df9-435e-91e4-02e0a6731581-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:03.538788 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.538673 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0eb10328-3df9-435e-91e4-02e0a6731581-data-volume\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:03.538788 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.538726 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0eb10328-3df9-435e-91e4-02e0a6731581-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:03.538881 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.538807 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0eb10328-3df9-435e-91e4-02e0a6731581-crio-socket\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:03.538922 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.538882 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whd8t\" (UniqueName: \"kubernetes.io/projected/0eb10328-3df9-435e-91e4-02e0a6731581-kube-api-access-whd8t\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:03.544715 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.544674 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:03.640140 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.640039 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0eb10328-3df9-435e-91e4-02e0a6731581-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:03.640140 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.640091 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0eb10328-3df9-435e-91e4-02e0a6731581-data-volume\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:03.640390 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.640157 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0eb10328-3df9-435e-91e4-02e0a6731581-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:03.640390 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.640210 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0eb10328-3df9-435e-91e4-02e0a6731581-crio-socket\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:03.640390 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:03.640256 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 15:41:03.640390 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:03.640285 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 15:41:03.640390 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.640260 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whd8t\" (UniqueName: \"kubernetes.io/projected/0eb10328-3df9-435e-91e4-02e0a6731581-kube-api-access-whd8t\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:03.640390 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:03.640342 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eb10328-3df9-435e-91e4-02e0a6731581-insights-runtime-extractor-tls podName:0eb10328-3df9-435e-91e4-02e0a6731581 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:04.14032136 +0000 UTC m=+48.317965788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0eb10328-3df9-435e-91e4-02e0a6731581-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q4x4h" (UID: "0eb10328-3df9-435e-91e4-02e0a6731581") : secret "insights-runtime-extractor-tls" not found Jul 07 15:41:03.640390 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.640372 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0eb10328-3df9-435e-91e4-02e0a6731581-crio-socket\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:03.640730 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:03.640431 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0eb10328-3df9-435e-91e4-02e0a6731581-kube-rbac-proxy-cm podName:0eb10328-3df9-435e-91e4-02e0a6731581 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:04.14039954 +0000 UTC m=+48.318043968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0eb10328-3df9-435e-91e4-02e0a6731581-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q4x4h" (UID: "0eb10328-3df9-435e-91e4-02e0a6731581") : configmap "kube-rbac-proxy" not found Jul 07 15:41:03.640730 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.640519 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0eb10328-3df9-435e-91e4-02e0a6731581-data-volume\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:03.651604 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.651578 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whd8t\" (UniqueName: \"kubernetes.io/projected/0eb10328-3df9-435e-91e4-02e0a6731581-kube-api-access-whd8t\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:03.700098 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.700062 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/1.log" Jul 07 15:41:03.700568 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.700552 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/0.log" Jul 07 15:41:03.700650 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.700590 2557 generic.go:356] "Generic (PLEG): container finished" podID="d1fdbe6c-661b-4201-87f1-7ca6fe2b561b" containerID="b9e4bb8b84e1b34afd561ad4e2d855a5338cb0a868f780d72227e91fb37c9bc3" exitCode=255 Jul 07 15:41:03.700706 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.700682 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" event={"ID":"d1fdbe6c-661b-4201-87f1-7ca6fe2b561b","Type":"ContainerDied","Data":"b9e4bb8b84e1b34afd561ad4e2d855a5338cb0a868f780d72227e91fb37c9bc3"} Jul 07 15:41:03.700766 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.700750 2557 scope.go:117] "RemoveContainer" containerID="790cd4da8542369aaedcb0ac10c280a5a0a6611acda1069fb066af60069b0c95" Jul 07 15:41:03.701002 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.700976 2557 scope.go:117] "RemoveContainer" containerID="b9e4bb8b84e1b34afd561ad4e2d855a5338cb0a868f780d72227e91fb37c9bc3" Jul 07 15:41:03.701239 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:03.701218 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-dn2nk_openshift-console-operator(d1fdbe6c-661b-4201-87f1-7ca6fe2b561b)\"" pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" podUID="d1fdbe6c-661b-4201-87f1-7ca6fe2b561b" Jul 07 15:41:03.704077 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.704041 2557 generic.go:356] "Generic (PLEG): container finished" podID="7d4917d4-c3ba-432c-83b2-0672de4a70f8" containerID="1dc403ee9074da389cc3a2377124dfb72dc0ccedee112ccbaa6a084401a49738" exitCode=0 Jul 07 15:41:03.704077 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.704067 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2l5gl" event={"ID":"7d4917d4-c3ba-432c-83b2-0672de4a70f8","Type":"ContainerDied","Data":"1dc403ee9074da389cc3a2377124dfb72dc0ccedee112ccbaa6a084401a49738"} Jul 07 15:41:03.872703 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.872662 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:03.893075 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.892984 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-p487q"] Jul 07 15:41:03.915847 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.915807 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-p487q"] Jul 07 15:41:03.916030 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.915983 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-p487q" Jul 07 15:41:03.918290 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.918266 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 07 15:41:03.918454 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.918269 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 07 15:41:03.918500 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.918478 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vvpr7\"" Jul 07 15:41:03.943530 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.943501 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ltl7\" (UniqueName: \"kubernetes.io/projected/8148a32a-727e-4d27-9cc5-43b0b3a8f89c-kube-api-access-5ltl7\") pod \"migrator-d57d58cc9-p487q\" (UID: \"8148a32a-727e-4d27-9cc5-43b0b3a8f89c\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-p487q" Jul 07 15:41:03.947279 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:03.947247 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:04.006464 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:04.006421 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:04.044800 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:04.044768 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ltl7\" (UniqueName: \"kubernetes.io/projected/8148a32a-727e-4d27-9cc5-43b0b3a8f89c-kube-api-access-5ltl7\") pod \"migrator-d57d58cc9-p487q\" (UID: \"8148a32a-727e-4d27-9cc5-43b0b3a8f89c\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-p487q" Jul 07 15:41:04.053931 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:04.053896 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ltl7\" (UniqueName: \"kubernetes.io/projected/8148a32a-727e-4d27-9cc5-43b0b3a8f89c-kube-api-access-5ltl7\") pod \"migrator-d57d58cc9-p487q\" (UID: \"8148a32a-727e-4d27-9cc5-43b0b3a8f89c\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-p487q" Jul 07 15:41:04.145907 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:04.145825 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0eb10328-3df9-435e-91e4-02e0a6731581-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:04.146296 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:04.145958 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 15:41:04.146296 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:04.145998 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0eb10328-3df9-435e-91e4-02e0a6731581-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:04.146296 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:04.146041 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0eb10328-3df9-435e-91e4-02e0a6731581-kube-rbac-proxy-cm podName:0eb10328-3df9-435e-91e4-02e0a6731581 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:05.146021434 +0000 UTC m=+49.323665881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0eb10328-3df9-435e-91e4-02e0a6731581-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q4x4h" (UID: "0eb10328-3df9-435e-91e4-02e0a6731581") : configmap "kube-rbac-proxy" not found Jul 07 15:41:04.146296 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:04.146098 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 15:41:04.146296 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:04.146174 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eb10328-3df9-435e-91e4-02e0a6731581-insights-runtime-extractor-tls podName:0eb10328-3df9-435e-91e4-02e0a6731581 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:05.146160226 +0000 UTC m=+49.323804656 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0eb10328-3df9-435e-91e4-02e0a6731581-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q4x4h" (UID: "0eb10328-3df9-435e-91e4-02e0a6731581") : secret "insights-runtime-extractor-tls" not found Jul 07 15:41:04.225487 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:04.225444 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-p487q" Jul 07 15:41:04.375300 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:04.375265 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-p487q"] Jul 07 15:41:04.378710 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:41:04.378683 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8148a32a_727e_4d27_9cc5_43b0b3a8f89c.slice/crio-6a6da432a2c3beee17f403a0d1e18a59c6d161206d4b85d0c07199bd81a0f251 WatchSource:0}: Error finding container 6a6da432a2c3beee17f403a0d1e18a59c6d161206d4b85d0c07199bd81a0f251: Status 404 returned error can't find the container with id 6a6da432a2c3beee17f403a0d1e18a59c6d161206d4b85d0c07199bd81a0f251 Jul 07 15:41:04.685391 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:04.685362 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lhvjn_e7b29c32-fe48-437a-956a-f4b0d4eab6c3/dns-node-resolver/0.log" Jul 07 15:41:04.708758 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:04.708723 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-p487q" event={"ID":"8148a32a-727e-4d27-9cc5-43b0b3a8f89c","Type":"ContainerStarted","Data":"6a6da432a2c3beee17f403a0d1e18a59c6d161206d4b85d0c07199bd81a0f251"} Jul 07 15:41:04.711872 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:04.711839 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2l5gl" event={"ID":"7d4917d4-c3ba-432c-83b2-0672de4a70f8","Type":"ContainerStarted","Data":"920024ba7e29e1ef52a1d52868cb2d9d9b7fbaf856b3b49285afc116f021daf2"} Jul 07 15:41:04.713204 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:04.713185 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/1.log" Jul 07 15:41:04.713453 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:04.713441 2557 scope.go:117] "RemoveContainer" containerID="b9e4bb8b84e1b34afd561ad4e2d855a5338cb0a868f780d72227e91fb37c9bc3" Jul 07 15:41:04.713594 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:04.713577 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-dn2nk_openshift-console-operator(d1fdbe6c-661b-4201-87f1-7ca6fe2b561b)\"" pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" podUID="d1fdbe6c-661b-4201-87f1-7ca6fe2b561b" Jul 07 15:41:04.734909 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:04.734850 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2l5gl" podStartSLOduration=7.995575881 podStartE2EDuration="47.734832058s" totalImagesPullingTime="39.739256177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:40:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:04.73409165 +0000 UTC m=+48.911736094" watchObservedRunningTime="2026-07-07 15:41:04.734832058 +0000 UTC m=+48.912476513" Jul 07 15:41:04.864057 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:04.864022 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:04.938238 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:04.938139 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:05.156353 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.156315 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0eb10328-3df9-435e-91e4-02e0a6731581-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:05.156838 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.156400 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0eb10328-3df9-435e-91e4-02e0a6731581-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:05.156838 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:05.156485 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 15:41:05.156838 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:05.156501 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 15:41:05.156838 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:05.156571 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eb10328-3df9-435e-91e4-02e0a6731581-insights-runtime-extractor-tls podName:0eb10328-3df9-435e-91e4-02e0a6731581 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:07.15655006 +0000 UTC m=+51.334194504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0eb10328-3df9-435e-91e4-02e0a6731581-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q4x4h" (UID: "0eb10328-3df9-435e-91e4-02e0a6731581") : secret "insights-runtime-extractor-tls" not found Jul 07 15:41:05.156838 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:05.156596 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0eb10328-3df9-435e-91e4-02e0a6731581-kube-rbac-proxy-cm podName:0eb10328-3df9-435e-91e4-02e0a6731581 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:07.156582997 +0000 UTC m=+51.334227428 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0eb10328-3df9-435e-91e4-02e0a6731581-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q4x4h" (UID: "0eb10328-3df9-435e-91e4-02e0a6731581") : configmap "kube-rbac-proxy" not found Jul 07 15:41:05.395237 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.395206 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-cwswx"] Jul 07 15:41:05.427424 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.427390 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-cwswx"] Jul 07 15:41:05.427586 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.427512 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-cwswx" Jul 07 15:41:05.429677 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.429655 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 07 15:41:05.430189 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.430164 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 07 15:41:05.430189 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.430178 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 07 15:41:05.430419 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.430249 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 07 15:41:05.430419 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.430250 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-g9hlp\"" Jul 07 15:41:05.460392 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.460365 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/01ccd175-0788-41f2-8903-0940ff2f3dc5-signing-cabundle\") pod \"service-ca-758f599c5-cwswx\" (UID: \"01ccd175-0788-41f2-8903-0940ff2f3dc5\") " pod="openshift-service-ca/service-ca-758f599c5-cwswx" Jul 07 15:41:05.460550 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.460449 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfh5h\" (UniqueName: \"kubernetes.io/projected/01ccd175-0788-41f2-8903-0940ff2f3dc5-kube-api-access-sfh5h\") pod \"service-ca-758f599c5-cwswx\" (UID: \"01ccd175-0788-41f2-8903-0940ff2f3dc5\") " pod="openshift-service-ca/service-ca-758f599c5-cwswx" Jul 07 15:41:05.460609 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.460552 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/01ccd175-0788-41f2-8903-0940ff2f3dc5-signing-key\") pod \"service-ca-758f599c5-cwswx\" (UID: \"01ccd175-0788-41f2-8903-0940ff2f3dc5\") " pod="openshift-service-ca/service-ca-758f599c5-cwswx" Jul 07 15:41:05.501675 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.501639 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:05.561552 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.561516 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfh5h\" (UniqueName: \"kubernetes.io/projected/01ccd175-0788-41f2-8903-0940ff2f3dc5-kube-api-access-sfh5h\") pod \"service-ca-758f599c5-cwswx\" (UID: \"01ccd175-0788-41f2-8903-0940ff2f3dc5\") " pod="openshift-service-ca/service-ca-758f599c5-cwswx" Jul 07 15:41:05.561708 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.561578 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/01ccd175-0788-41f2-8903-0940ff2f3dc5-signing-key\") pod \"service-ca-758f599c5-cwswx\" (UID: \"01ccd175-0788-41f2-8903-0940ff2f3dc5\") " pod="openshift-service-ca/service-ca-758f599c5-cwswx" Jul 07 15:41:05.561839 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.561814 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/01ccd175-0788-41f2-8903-0940ff2f3dc5-signing-cabundle\") pod \"service-ca-758f599c5-cwswx\" (UID: \"01ccd175-0788-41f2-8903-0940ff2f3dc5\") " pod="openshift-service-ca/service-ca-758f599c5-cwswx" Jul 07 15:41:05.562391 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.562369 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/01ccd175-0788-41f2-8903-0940ff2f3dc5-signing-cabundle\") pod \"service-ca-758f599c5-cwswx\" (UID: \"01ccd175-0788-41f2-8903-0940ff2f3dc5\") " pod="openshift-service-ca/service-ca-758f599c5-cwswx" Jul 07 15:41:05.563840 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.563818 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/01ccd175-0788-41f2-8903-0940ff2f3dc5-signing-key\") pod \"service-ca-758f599c5-cwswx\" (UID: \"01ccd175-0788-41f2-8903-0940ff2f3dc5\") " pod="openshift-service-ca/service-ca-758f599c5-cwswx" Jul 07 15:41:05.570185 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.570156 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfh5h\" (UniqueName: \"kubernetes.io/projected/01ccd175-0788-41f2-8903-0940ff2f3dc5-kube-api-access-sfh5h\") pod \"service-ca-758f599c5-cwswx\" (UID: \"01ccd175-0788-41f2-8903-0940ff2f3dc5\") " pod="openshift-service-ca/service-ca-758f599c5-cwswx" Jul 07 15:41:05.593963 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.593935 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:05.738184 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.738082 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-cwswx" Jul 07 15:41:05.885803 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:05.885772 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mcrlt_78ecda82-54b3-4727-89b9-5d55dd0ca428/node-ca/0.log" Jul 07 15:41:06.103639 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:06.103614 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-cwswx"] Jul 07 15:41:06.106366 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:41:06.106343 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ccd175_0788_41f2_8903_0940ff2f3dc5.slice/crio-d81bac374e6370a49b058fdc9c7b6dd2ec1e699a3db89bbcd8345cf20194d704 WatchSource:0}: Error finding container d81bac374e6370a49b058fdc9c7b6dd2ec1e699a3db89bbcd8345cf20194d704: Status 404 returned error can't find the container with id d81bac374e6370a49b058fdc9c7b6dd2ec1e699a3db89bbcd8345cf20194d704 Jul 07 15:41:06.284793 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:06.284764 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rk7b5_a9aad135-443d-43c6-8b9d-94107c57ef21/serve-healthcheck-canary/0.log" Jul 07 15:41:06.722965 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:06.722854 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-cwswx" event={"ID":"01ccd175-0788-41f2-8903-0940ff2f3dc5","Type":"ContainerStarted","Data":"604ef08f689b5e8f94d6841668d403309611d5cb12a1a976f10049427b4c0ac6"} Jul 07 15:41:06.722965 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:06.722914 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-cwswx" event={"ID":"01ccd175-0788-41f2-8903-0940ff2f3dc5","Type":"ContainerStarted","Data":"d81bac374e6370a49b058fdc9c7b6dd2ec1e699a3db89bbcd8345cf20194d704"} Jul 07 15:41:06.724649 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:06.724625 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-p487q" event={"ID":"8148a32a-727e-4d27-9cc5-43b0b3a8f89c","Type":"ContainerStarted","Data":"6bf1ad65e68d287a4cad2fa117a0491db693dffea8dbc5889c448387cb461657"} Jul 07 15:41:06.724761 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:06.724668 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-p487q" event={"ID":"8148a32a-727e-4d27-9cc5-43b0b3a8f89c","Type":"ContainerStarted","Data":"a5059df40da5fa793a19ddc8c0921ba7fe26fd734d7d0ea180162304fb2988bf"} Jul 07 15:41:06.742844 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:06.742783 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-cwswx" podStartSLOduration=1.742761399 podStartE2EDuration="1.742761399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:06.74186294 +0000 UTC m=+50.919507391" watchObservedRunningTime="2026-07-07 15:41:06.742761399 +0000 UTC m=+50.920405851" Jul 07 15:41:06.766421 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:06.766367 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-p487q" podStartSLOduration=2.182986687 podStartE2EDuration="3.766351291s" totalImagesPullingTime="1.583364604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:06.764099008 +0000 UTC m=+50.941743460" watchObservedRunningTime="2026-07-07 15:41:06.766351291 +0000 UTC m=+50.943995741" Jul 07 15:41:07.176873 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:07.176830 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0eb10328-3df9-435e-91e4-02e0a6731581-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:07.177071 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:07.176898 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0eb10328-3df9-435e-91e4-02e0a6731581-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:07.177071 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:07.176955 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 15:41:07.177071 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:07.176985 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 15:41:07.177071 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:07.177002 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0eb10328-3df9-435e-91e4-02e0a6731581-kube-rbac-proxy-cm podName:0eb10328-3df9-435e-91e4-02e0a6731581 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:11.17698875 +0000 UTC m=+55.354633178 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0eb10328-3df9-435e-91e4-02e0a6731581-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q4x4h" (UID: "0eb10328-3df9-435e-91e4-02e0a6731581") : configmap "kube-rbac-proxy" not found Jul 07 15:41:07.177071 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:07.177048 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eb10328-3df9-435e-91e4-02e0a6731581-insights-runtime-extractor-tls podName:0eb10328-3df9-435e-91e4-02e0a6731581 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:11.177025003 +0000 UTC m=+55.354669430 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0eb10328-3df9-435e-91e4-02e0a6731581-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q4x4h" (UID: "0eb10328-3df9-435e-91e4-02e0a6731581") : secret "insights-runtime-extractor-tls" not found Jul 07 15:41:08.086452 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:08.086415 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-metrics-certs\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:41:08.086926 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:08.086460 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-metrics-tls\") pod \"dns-default-spwqp\" (UID: \"4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3\") " pod="openshift-dns/dns-default-spwqp" Jul 07 15:41:08.086926 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:08.086514 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8mf6j\" (UID: \"cdd40d24-e30b-496c-b92b-7c80e6ee0b33\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j" Jul 07 15:41:08.086926 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:08.086567 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 15:41:08.086926 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:08.086637 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-metrics-certs podName:6937108d-00d9-456e-80a3-5230106a0708 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:24.086622558 +0000 UTC m=+68.264266985 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-metrics-certs") pod "router-default-7b5cfc467-zp2s8" (UID: "6937108d-00d9-456e-80a3-5230106a0708") : secret "router-metrics-certs-default" not found Jul 07 15:41:08.086926 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:08.086641 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 15:41:08.086926 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:08.086574 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:41:08.086926 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:08.086663 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 15:41:08.086926 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:08.086644 2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 15:41:08.086926 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:08.086696 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-metrics-tls podName:4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:24.086681788 +0000 UTC m=+68.264326234 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-metrics-tls") pod "dns-default-spwqp" (UID: "4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3") : secret "dns-default-metrics-tls" not found Jul 07 15:41:08.086926 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:08.086715 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6937108d-00d9-456e-80a3-5230106a0708-service-ca-bundle\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:41:08.086926 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:08.086726 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-cluster-monitoring-operator-tls podName:cdd40d24-e30b-496c-b92b-7c80e6ee0b33 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:24.086708231 +0000 UTC m=+68.264352676 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8mf6j" (UID: "cdd40d24-e30b-496c-b92b-7c80e6ee0b33") : secret "cluster-monitoring-operator-tls" not found Jul 07 15:41:08.086926 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:08.086696 2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-848545c49c-thmhm: secret "image-registry-tls" not found Jul 07 15:41:08.086926 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:08.086768 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-jrjtw\" (UID: \"8f635da8-3f7f-4450-bdc5-bb8616c2fb12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jrjtw" Jul 07 15:41:08.086926 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:08.086777 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6937108d-00d9-456e-80a3-5230106a0708-service-ca-bundle podName:6937108d-00d9-456e-80a3-5230106a0708 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:24.086766357 +0000 UTC m=+68.264410802 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6937108d-00d9-456e-80a3-5230106a0708-service-ca-bundle") pod "router-default-7b5cfc467-zp2s8" (UID: "6937108d-00d9-456e-80a3-5230106a0708") : configmap references non-existent config key: service-ca.crt Jul 07 15:41:08.086926 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:08.086794 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls podName:ce7245b7-e2e6-43df-9ae2-568fb739ed49 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:24.08678592 +0000 UTC m=+68.264430357 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls") pod "image-registry-848545c49c-thmhm" (UID: "ce7245b7-e2e6-43df-9ae2-568fb739ed49") : secret "image-registry-tls" not found Jul 07 15:41:08.086926 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:08.086864 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 15:41:08.087734 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:08.086904 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-samples-operator-tls podName:8f635da8-3f7f-4450-bdc5-bb8616c2fb12 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:24.08689406 +0000 UTC m=+68.264538494 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-jrjtw" (UID: "8f635da8-3f7f-4450-bdc5-bb8616c2fb12") : secret "samples-operator-tls" not found Jul 07 15:41:08.187482 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:08.187446 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a278ee42-38b2-4aee-b359-602e3bbf59be-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-vfwk5\" (UID: \"a278ee42-38b2-4aee-b359-602e3bbf59be\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-vfwk5" Jul 07 15:41:08.187660 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:08.187597 2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 15:41:08.187701 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:08.187663 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a278ee42-38b2-4aee-b359-602e3bbf59be-networking-console-plugin-cert podName:a278ee42-38b2-4aee-b359-602e3bbf59be nodeName:}" failed. No retries permitted until 2026-07-07 15:41:24.187649213 +0000 UTC m=+68.365293641 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a278ee42-38b2-4aee-b359-602e3bbf59be-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-vfwk5" (UID: "a278ee42-38b2-4aee-b359-602e3bbf59be") : secret "networking-console-plugin-cert" not found Jul 07 15:41:11.215531 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:11.215487 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0eb10328-3df9-435e-91e4-02e0a6731581-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:11.216092 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:11.215561 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0eb10328-3df9-435e-91e4-02e0a6731581-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:11.216092 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:11.215648 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 15:41:11.216092 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:11.215715 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eb10328-3df9-435e-91e4-02e0a6731581-insights-runtime-extractor-tls podName:0eb10328-3df9-435e-91e4-02e0a6731581 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:19.215699554 +0000 UTC m=+63.393343981 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0eb10328-3df9-435e-91e4-02e0a6731581-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q4x4h" (UID: "0eb10328-3df9-435e-91e4-02e0a6731581") : secret "insights-runtime-extractor-tls" not found Jul 07 15:41:11.216092 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:11.216083 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0eb10328-3df9-435e-91e4-02e0a6731581-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:12.619724 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:12.619678 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" Jul 07 15:41:12.620095 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:12.619752 2557 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" Jul 07 15:41:12.620170 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:12.620105 2557 scope.go:117] "RemoveContainer" containerID="b9e4bb8b84e1b34afd561ad4e2d855a5338cb0a868f780d72227e91fb37c9bc3" Jul 07 15:41:12.620302 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:41:12.620281 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-dn2nk_openshift-console-operator(d1fdbe6c-661b-4201-87f1-7ca6fe2b561b)\"" pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" podUID="d1fdbe6c-661b-4201-87f1-7ca6fe2b561b" Jul 07 15:41:17.622463 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:17.622426 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hkj2z" Jul 07 15:41:19.291680 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:19.291643 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0eb10328-3df9-435e-91e4-02e0a6731581-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:19.293950 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:19.293928 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0eb10328-3df9-435e-91e4-02e0a6731581-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4x4h\" (UID: \"0eb10328-3df9-435e-91e4-02e0a6731581\") " pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:19.416103 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:19.416072 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fwwjf\"" Jul 07 15:41:19.424747 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:19.424728 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q4x4h" Jul 07 15:41:19.563434 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:19.563356 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q4x4h"] Jul 07 15:41:19.566489 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:41:19.566465 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb10328_3df9_435e_91e4_02e0a6731581.slice/crio-076cbaa5008ed3f7c396887bd7c54e287871280c83c95e71a95b2e1099b6d44c WatchSource:0}: Error finding container 076cbaa5008ed3f7c396887bd7c54e287871280c83c95e71a95b2e1099b6d44c: Status 404 returned error can't find the container with id 076cbaa5008ed3f7c396887bd7c54e287871280c83c95e71a95b2e1099b6d44c Jul 07 15:41:19.779685 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:19.779654 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q4x4h" event={"ID":"0eb10328-3df9-435e-91e4-02e0a6731581","Type":"ContainerStarted","Data":"076cbaa5008ed3f7c396887bd7c54e287871280c83c95e71a95b2e1099b6d44c"} Jul 07 15:41:22.017293 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:22.017236 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret\") pod \"global-pull-secret-syncer-q2m8b\" (UID: \"b99c24f7-0333-47fe-8b1a-ed9c6a10d23c\") " pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:41:22.017293 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:22.017294 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs\") pod \"network-metrics-daemon-tb5cw\" (UID: \"285707f9-4f1b-480e-ae8e-123d5ba608c6\") " pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:41:22.020213 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:22.019714 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 15:41:22.020213 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:22.020034 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 15:41:22.029879 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:22.029847 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b99c24f7-0333-47fe-8b1a-ed9c6a10d23c-original-pull-secret\") pod \"global-pull-secret-syncer-q2m8b\" (UID: \"b99c24f7-0333-47fe-8b1a-ed9c6a10d23c\") " pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:41:22.030019 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:22.029911 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285707f9-4f1b-480e-ae8e-123d5ba608c6-metrics-certs\") pod \"network-metrics-daemon-tb5cw\" (UID: \"285707f9-4f1b-480e-ae8e-123d5ba608c6\") " pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:41:22.057301 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:22.057262 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jh4tr\"" Jul 07 15:41:22.066521 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:22.066089 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5cw" Jul 07 15:41:22.092928 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:22.092899 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2m8b" Jul 07 15:41:22.219816 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:22.219780 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2klg\" (UniqueName: \"kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg\") pod \"network-check-target-hwwk7\" (UID: \"a767f075-0967-445e-8038-d22a7eda5b15\") " pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:41:22.222246 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:22.222216 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2klg\" (UniqueName: \"kubernetes.io/projected/a767f075-0967-445e-8038-d22a7eda5b15-kube-api-access-m2klg\") pod \"network-check-target-hwwk7\" (UID: \"a767f075-0967-445e-8038-d22a7eda5b15\") " pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:41:22.385807 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:22.385778 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v8g69\"" Jul 07 15:41:22.394257 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:22.394233 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:41:23.332715 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:23.332295 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q2m8b"] Jul 07 15:41:23.335748 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:41:23.335722 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99c24f7_0333_47fe_8b1a_ed9c6a10d23c.slice/crio-3362e032be62b93b65209a4e6cfcf450256af7e792cc91170860dba8e3b7e025 WatchSource:0}: Error finding container 3362e032be62b93b65209a4e6cfcf450256af7e792cc91170860dba8e3b7e025: Status 404 returned error can't find the container with id 3362e032be62b93b65209a4e6cfcf450256af7e792cc91170860dba8e3b7e025 Jul 07 15:41:23.554944 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:23.554913 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tb5cw"] Jul 07 15:41:23.557710 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:23.557683 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hwwk7"] Jul 07 15:41:23.559504 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:41:23.559480 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285707f9_4f1b_480e_ae8e_123d5ba608c6.slice/crio-cf13e49adbc3a7a97dc2aee97b53523d0451380b844bb50313464b72bee1c3b9 WatchSource:0}: Error finding container cf13e49adbc3a7a97dc2aee97b53523d0451380b844bb50313464b72bee1c3b9: Status 404 returned error can't find the container with id cf13e49adbc3a7a97dc2aee97b53523d0451380b844bb50313464b72bee1c3b9 Jul 07 15:41:23.562386 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:41:23.562366 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda767f075_0967_445e_8038_d22a7eda5b15.slice/crio-febc8d0f22513da9a9b304e71b2072df2199a8140efae7f0237bb7904281e47e WatchSource:0}: Error finding container febc8d0f22513da9a9b304e71b2072df2199a8140efae7f0237bb7904281e47e: Status 404 returned error can't find the container with id febc8d0f22513da9a9b304e71b2072df2199a8140efae7f0237bb7904281e47e Jul 07 15:41:23.796557 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:23.796505 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q2m8b" event={"ID":"b99c24f7-0333-47fe-8b1a-ed9c6a10d23c","Type":"ContainerStarted","Data":"3362e032be62b93b65209a4e6cfcf450256af7e792cc91170860dba8e3b7e025"} Jul 07 15:41:23.798283 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:23.798254 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hwwk7" event={"ID":"a767f075-0967-445e-8038-d22a7eda5b15","Type":"ContainerStarted","Data":"749452eaf6295a67616aa1c3d86c323c00216ea5755f4ecb1e70c110af369fb6"} Jul 07 15:41:23.798429 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:23.798303 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hwwk7" event={"ID":"a767f075-0967-445e-8038-d22a7eda5b15","Type":"ContainerStarted","Data":"febc8d0f22513da9a9b304e71b2072df2199a8140efae7f0237bb7904281e47e"} Jul 07 15:41:23.798429 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:23.798345 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:41:23.800076 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:23.800054 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q4x4h" event={"ID":"0eb10328-3df9-435e-91e4-02e0a6731581","Type":"ContainerStarted","Data":"41cde1c6484aac3fbc3e106bffcf9fca4f69df710c5c032c363d807a8cf5bf23"} Jul 07 15:41:23.801377 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:23.801355 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tb5cw" event={"ID":"285707f9-4f1b-480e-ae8e-123d5ba608c6","Type":"ContainerStarted","Data":"cf13e49adbc3a7a97dc2aee97b53523d0451380b844bb50313464b72bee1c3b9"} Jul 07 15:41:23.813335 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:23.813171 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hwwk7" podStartSLOduration=66.813153502 podStartE2EDuration="1m6.813153502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:23.813090897 +0000 UTC m=+67.990735328" watchObservedRunningTime="2026-07-07 15:41:23.813153502 +0000 UTC m=+67.990797957" Jul 07 15:41:24.138862 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.138836 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-metrics-certs\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:41:24.138980 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.138884 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-metrics-tls\") pod \"dns-default-spwqp\" (UID: \"4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3\") " pod="openshift-dns/dns-default-spwqp" Jul 07 15:41:24.138980 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.138948 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8mf6j\" (UID: \"cdd40d24-e30b-496c-b92b-7c80e6ee0b33\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j" Jul 07 15:41:24.139091 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.139008 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:41:24.139091 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.139040 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6937108d-00d9-456e-80a3-5230106a0708-service-ca-bundle\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:41:24.139091 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.139068 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-jrjtw\" (UID: \"8f635da8-3f7f-4450-bdc5-bb8616c2fb12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jrjtw" Jul 07 15:41:24.139899 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.139849 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6937108d-00d9-456e-80a3-5230106a0708-service-ca-bundle\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:41:24.141761 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.141714 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6937108d-00d9-456e-80a3-5230106a0708-metrics-certs\") pod \"router-default-7b5cfc467-zp2s8\" (UID: \"6937108d-00d9-456e-80a3-5230106a0708\") " pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:41:24.141935 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.141910 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3-metrics-tls\") pod \"dns-default-spwqp\" (UID: \"4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3\") " pod="openshift-dns/dns-default-spwqp" Jul 07 15:41:24.142021 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.141919 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdd40d24-e30b-496c-b92b-7c80e6ee0b33-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8mf6j\" (UID: \"cdd40d24-e30b-496c-b92b-7c80e6ee0b33\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j" Jul 07 15:41:24.142350 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.142328 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f635da8-3f7f-4450-bdc5-bb8616c2fb12-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-jrjtw\" (UID: \"8f635da8-3f7f-4450-bdc5-bb8616c2fb12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jrjtw" Jul 07 15:41:24.143070 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.143035 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls\") pod \"image-registry-848545c49c-thmhm\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:41:24.239863 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.239832 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a278ee42-38b2-4aee-b359-602e3bbf59be-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-vfwk5\" (UID: \"a278ee42-38b2-4aee-b359-602e3bbf59be\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-vfwk5" Jul 07 15:41:24.242303 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.242278 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a278ee42-38b2-4aee-b359-602e3bbf59be-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-vfwk5\" (UID: \"a278ee42-38b2-4aee-b359-602e3bbf59be\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-vfwk5" Jul 07 15:41:24.294877 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.294687 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kdgps\"" Jul 07 15:41:24.303485 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.303447 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:41:24.333843 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.333637 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-kk95r\"" Jul 07 15:41:24.341000 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.340973 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jrjtw" Jul 07 15:41:24.346004 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.345667 2557 scope.go:117] "RemoveContainer" containerID="b9e4bb8b84e1b34afd561ad4e2d855a5338cb0a868f780d72227e91fb37c9bc3" Jul 07 15:41:24.365462 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.365404 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-q6mbt\"" Jul 07 15:41:24.375485 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.374464 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:41:24.381323 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.380160 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tn46g\"" Jul 07 15:41:24.388741 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.388365 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4gzbc\"" Jul 07 15:41:24.389136 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.388928 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-spwqp" Jul 07 15:41:24.396155 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.396077 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j" Jul 07 15:41:24.415910 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.415683 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9w2h6\"" Jul 07 15:41:24.424267 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.424175 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-vfwk5" Jul 07 15:41:24.500838 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.500642 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-848545c49c-thmhm"] Jul 07 15:41:24.510243 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:41:24.509343 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7245b7_e2e6_43df_9ae2_568fb739ed49.slice/crio-dd2fec7ada1f7dcb777f39738a572ee350d7c8dffb5d89faf92112747ce4049c WatchSource:0}: Error finding container dd2fec7ada1f7dcb777f39738a572ee350d7c8dffb5d89faf92112747ce4049c: Status 404 returned error can't find the container with id dd2fec7ada1f7dcb777f39738a572ee350d7c8dffb5d89faf92112747ce4049c Jul 07 15:41:24.587475 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.586457 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jrjtw"] Jul 07 15:41:24.627824 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.627762 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7b5cfc467-zp2s8"] Jul 07 15:41:24.654277 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.654202 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-spwqp"] Jul 07 15:41:24.658902 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:41:24.658864 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eda6cd2_bb41_4f2e_8c1e_1447a69b29a3.slice/crio-14dfbadd7c9b86c3ee98fd436e2c45def3f44eab493c62961ae70b8916abac71 WatchSource:0}: Error finding container 14dfbadd7c9b86c3ee98fd436e2c45def3f44eab493c62961ae70b8916abac71: Status 404 returned error can't find the container with id 14dfbadd7c9b86c3ee98fd436e2c45def3f44eab493c62961ae70b8916abac71 Jul 07 15:41:24.688461 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.688326 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j"] Jul 07 15:41:24.691870 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:41:24.691833 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd40d24_e30b_496c_b92b_7c80e6ee0b33.slice/crio-0c454ff757b1f8b31c90941d5f59a7b623a9570c1efb545ac6aacea0bea3ad0d WatchSource:0}: Error finding container 0c454ff757b1f8b31c90941d5f59a7b623a9570c1efb545ac6aacea0bea3ad0d: Status 404 returned error can't find the container with id 0c454ff757b1f8b31c90941d5f59a7b623a9570c1efb545ac6aacea0bea3ad0d Jul 07 15:41:24.810236 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.810160 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-spwqp" event={"ID":"4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3","Type":"ContainerStarted","Data":"14dfbadd7c9b86c3ee98fd436e2c45def3f44eab493c62961ae70b8916abac71"} Jul 07 15:41:24.813172 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.813146 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/1.log" Jul 07 15:41:24.813294 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.813235 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" event={"ID":"d1fdbe6c-661b-4201-87f1-7ca6fe2b561b","Type":"ContainerStarted","Data":"7bd5479b9ea9821ded4e368e45ce6565cd870f174c5fba9399e2eb4a74d5a154"} Jul 07 15:41:24.813754 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.813735 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" Jul 07 15:41:24.816541 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.816512 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7b5cfc467-zp2s8" event={"ID":"6937108d-00d9-456e-80a3-5230106a0708","Type":"ContainerStarted","Data":"aad38be9e8c8dbccd63d096c3c6976523e4677be341b8bf17fcf74ab353cc9c5"} Jul 07 15:41:24.816641 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.816560 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7b5cfc467-zp2s8" event={"ID":"6937108d-00d9-456e-80a3-5230106a0708","Type":"ContainerStarted","Data":"24b6cd16d21a30c008ba2a662759adcaaf522d165d2db2a9d02a2b88bf4d58b3"} Jul 07 15:41:24.819026 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.819000 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-848545c49c-thmhm" event={"ID":"ce7245b7-e2e6-43df-9ae2-568fb739ed49","Type":"ContainerStarted","Data":"b03e154d91193275dcc9be575eeefcfc3a869605e7561f9f5d9c242031f32f1e"} Jul 07 15:41:24.819152 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.819047 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-848545c49c-thmhm" event={"ID":"ce7245b7-e2e6-43df-9ae2-568fb739ed49","Type":"ContainerStarted","Data":"dd2fec7ada1f7dcb777f39738a572ee350d7c8dffb5d89faf92112747ce4049c"} Jul 07 15:41:24.819216 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.819156 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:41:24.821253 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.821229 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q4x4h" event={"ID":"0eb10328-3df9-435e-91e4-02e0a6731581","Type":"ContainerStarted","Data":"c46830639ed744205667510255544b625f33e266597dbc755ada21bdbf563cae"} Jul 07 15:41:24.822632 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.822592 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j" event={"ID":"cdd40d24-e30b-496c-b92b-7c80e6ee0b33","Type":"ContainerStarted","Data":"0c454ff757b1f8b31c90941d5f59a7b623a9570c1efb545ac6aacea0bea3ad0d"} Jul 07 15:41:24.824768 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.824744 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jrjtw" event={"ID":"8f635da8-3f7f-4450-bdc5-bb8616c2fb12","Type":"ContainerStarted","Data":"bd162c7d2446527acefdf0d5f84ce086a9b4f1ff9e4bc4e9200f54a91e6861c0"} Jul 07 15:41:24.833788 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.833042 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" podStartSLOduration=55.973268155 podStartE2EDuration="1m4.833024274s" totalImagesPullingTime="8.859756119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:24.83254381 +0000 UTC m=+69.010188287" watchObservedRunningTime="2026-07-07 15:41:24.833024274 +0000 UTC m=+69.010668726" Jul 07 15:41:24.885155 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.882027 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-848545c49c-thmhm" podStartSLOduration=67.882003251 podStartE2EDuration="1m7.882003251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:24.854100795 +0000 UTC m=+69.031745246" watchObservedRunningTime="2026-07-07 15:41:24.882003251 +0000 UTC m=+69.059647699" Jul 07 15:41:24.892099 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.890853 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7b5cfc467-zp2s8" podStartSLOduration=64.890833515 podStartE2EDuration="1m4.890833515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:24.881197107 +0000 UTC m=+69.058841547" watchObservedRunningTime="2026-07-07 15:41:24.890833515 +0000 UTC m=+69.068477965" Jul 07 15:41:24.892281 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:24.892262 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-vfwk5"] Jul 07 15:41:25.000168 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:25.000133 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-dn2nk" Jul 07 15:41:25.215369 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:41:25.215296 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda278ee42_38b2_4aee_b359_602e3bbf59be.slice/crio-c60efa40021478f9e3cb5f759a82105f4eefc01e3d445bd0a6c750711a8eb55b WatchSource:0}: Error finding container c60efa40021478f9e3cb5f759a82105f4eefc01e3d445bd0a6c750711a8eb55b: Status 404 returned error can't find the container with id c60efa40021478f9e3cb5f759a82105f4eefc01e3d445bd0a6c750711a8eb55b Jul 07 15:41:25.375221 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:25.375164 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:41:25.378778 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:25.378575 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:41:25.833338 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:25.833299 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tb5cw" event={"ID":"285707f9-4f1b-480e-ae8e-123d5ba608c6","Type":"ContainerStarted","Data":"1d497906d23275e5089110631809ec864a07cdbbf7cd82347906ded3c0062b1a"} Jul 07 15:41:25.833531 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:25.833370 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tb5cw" event={"ID":"285707f9-4f1b-480e-ae8e-123d5ba608c6","Type":"ContainerStarted","Data":"298229dc6a104e1bb3ae359c96813a452c6585976de4d501fc1565f82b8fa722"} Jul 07 15:41:25.837643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:25.836413 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-vfwk5" event={"ID":"a278ee42-38b2-4aee-b359-602e3bbf59be","Type":"ContainerStarted","Data":"c60efa40021478f9e3cb5f759a82105f4eefc01e3d445bd0a6c750711a8eb55b"} Jul 07 15:41:25.837643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:25.837461 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:41:25.838723 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:25.838703 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7b5cfc467-zp2s8" Jul 07 15:41:25.874031 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:25.873961 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tb5cw" podStartSLOduration=67.171977117 podStartE2EDuration="1m8.873942497s" totalImagesPullingTime="1.70196538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:25.854809023 +0000 UTC m=+70.032453475" watchObservedRunningTime="2026-07-07 15:41:25.873942497 +0000 UTC m=+70.051586948" Jul 07 15:41:27.066590 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:27.066552 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-6zs7q"] Jul 07 15:41:27.072765 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:27.071025 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-6zs7q" Jul 07 15:41:27.074023 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:27.073992 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 15:41:27.074179 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:27.073992 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-mk577\"" Jul 07 15:41:27.074367 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:27.074346 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 15:41:27.087304 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:27.087278 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-6zs7q"] Jul 07 15:41:27.173806 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:27.173772 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4lc6\" (UniqueName: \"kubernetes.io/projected/e26092ce-3f22-4a9e-b816-428098ee2569-kube-api-access-h4lc6\") pod \"downloads-685bfb5f64-6zs7q\" (UID: \"e26092ce-3f22-4a9e-b816-428098ee2569\") " pod="openshift-console/downloads-685bfb5f64-6zs7q" Jul 07 15:41:27.274695 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:27.274609 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4lc6\" (UniqueName: \"kubernetes.io/projected/e26092ce-3f22-4a9e-b816-428098ee2569-kube-api-access-h4lc6\") pod \"downloads-685bfb5f64-6zs7q\" (UID: \"e26092ce-3f22-4a9e-b816-428098ee2569\") " pod="openshift-console/downloads-685bfb5f64-6zs7q" Jul 07 15:41:27.289787 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:27.289751 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4lc6\" (UniqueName: \"kubernetes.io/projected/e26092ce-3f22-4a9e-b816-428098ee2569-kube-api-access-h4lc6\") pod \"downloads-685bfb5f64-6zs7q\" (UID: \"e26092ce-3f22-4a9e-b816-428098ee2569\") " pod="openshift-console/downloads-685bfb5f64-6zs7q" Jul 07 15:41:27.384366 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:27.384332 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-6zs7q" Jul 07 15:41:30.510351 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.510313 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74c996ddb-swsbz"] Jul 07 15:41:30.574607 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.574572 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74c996ddb-swsbz"] Jul 07 15:41:30.574803 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.574682 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:30.577207 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.577173 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-t87sb\"" Jul 07 15:41:30.577207 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.577173 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 15:41:30.577795 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.577695 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 15:41:30.577795 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.577753 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 15:41:30.577973 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.577700 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 15:41:30.577973 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.577918 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 15:41:30.705528 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.705491 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7338a0a-ea21-4f23-ae05-625d9b884463-console-serving-cert\") pod \"console-74c996ddb-swsbz\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:30.705717 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.705540 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpj4h\" (UniqueName: \"kubernetes.io/projected/a7338a0a-ea21-4f23-ae05-625d9b884463-kube-api-access-qpj4h\") pod \"console-74c996ddb-swsbz\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:30.705717 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.705626 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7338a0a-ea21-4f23-ae05-625d9b884463-console-config\") pod \"console-74c996ddb-swsbz\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:30.705717 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.705674 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7338a0a-ea21-4f23-ae05-625d9b884463-oauth-serving-cert\") pod \"console-74c996ddb-swsbz\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:30.705717 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.705706 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7338a0a-ea21-4f23-ae05-625d9b884463-console-oauth-config\") pod \"console-74c996ddb-swsbz\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:30.705931 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.705741 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7338a0a-ea21-4f23-ae05-625d9b884463-service-ca\") pod \"console-74c996ddb-swsbz\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:30.806630 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.806585 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7338a0a-ea21-4f23-ae05-625d9b884463-oauth-serving-cert\") pod \"console-74c996ddb-swsbz\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:30.806819 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.806658 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7338a0a-ea21-4f23-ae05-625d9b884463-console-oauth-config\") pod \"console-74c996ddb-swsbz\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:30.806819 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.806704 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7338a0a-ea21-4f23-ae05-625d9b884463-service-ca\") pod \"console-74c996ddb-swsbz\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:30.806819 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.806748 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7338a0a-ea21-4f23-ae05-625d9b884463-console-serving-cert\") pod \"console-74c996ddb-swsbz\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:30.806819 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.806782 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpj4h\" (UniqueName: \"kubernetes.io/projected/a7338a0a-ea21-4f23-ae05-625d9b884463-kube-api-access-qpj4h\") pod \"console-74c996ddb-swsbz\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:30.807044 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.806852 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7338a0a-ea21-4f23-ae05-625d9b884463-console-config\") pod \"console-74c996ddb-swsbz\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:30.807420 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.807378 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7338a0a-ea21-4f23-ae05-625d9b884463-oauth-serving-cert\") pod \"console-74c996ddb-swsbz\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:30.807540 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.807468 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7338a0a-ea21-4f23-ae05-625d9b884463-service-ca\") pod \"console-74c996ddb-swsbz\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:30.807603 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.807585 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7338a0a-ea21-4f23-ae05-625d9b884463-console-config\") pod \"console-74c996ddb-swsbz\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:30.809584 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.809560 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7338a0a-ea21-4f23-ae05-625d9b884463-console-serving-cert\") pod \"console-74c996ddb-swsbz\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:30.809711 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.809564 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7338a0a-ea21-4f23-ae05-625d9b884463-console-oauth-config\") pod \"console-74c996ddb-swsbz\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:30.815200 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.815175 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpj4h\" (UniqueName: \"kubernetes.io/projected/a7338a0a-ea21-4f23-ae05-625d9b884463-kube-api-access-qpj4h\") pod \"console-74c996ddb-swsbz\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:30.887664 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:30.887587 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:31.921762 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:31.921731 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-6zs7q"] Jul 07 15:41:31.927654 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:41:31.927625 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26092ce_3f22_4a9e_b816_428098ee2569.slice/crio-e4bb11c87d3878ce29bba6c33fbf37aeff56fc64225531f4dbbd474378dea568 WatchSource:0}: Error finding container e4bb11c87d3878ce29bba6c33fbf37aeff56fc64225531f4dbbd474378dea568: Status 404 returned error can't find the container with id e4bb11c87d3878ce29bba6c33fbf37aeff56fc64225531f4dbbd474378dea568 Jul 07 15:41:31.946527 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:31.946384 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74c996ddb-swsbz"] Jul 07 15:41:31.949831 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:41:31.949792 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7338a0a_ea21_4f23_ae05_625d9b884463.slice/crio-ec3ceddbd2ef2fa2b389a68b1062eb8ef25d325949e61903bcbcbff1dd3a4bdc WatchSource:0}: Error finding container ec3ceddbd2ef2fa2b389a68b1062eb8ef25d325949e61903bcbcbff1dd3a4bdc: Status 404 returned error can't find the container with id ec3ceddbd2ef2fa2b389a68b1062eb8ef25d325949e61903bcbcbff1dd3a4bdc Jul 07 15:41:32.387722 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.386659 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j7997"] Jul 07 15:41:32.399734 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.398853 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j7997" Jul 07 15:41:32.404420 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.401453 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-znmr2\"" Jul 07 15:41:32.404420 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.401681 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 07 15:41:32.404420 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.404239 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j7997"] Jul 07 15:41:32.524418 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.524383 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8518e838-f15a-4322-b246-5a439d4c6b62-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-j7997\" (UID: \"8518e838-f15a-4322-b246-5a439d4c6b62\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j7997" Jul 07 15:41:32.625670 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.625584 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8518e838-f15a-4322-b246-5a439d4c6b62-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-j7997\" (UID: \"8518e838-f15a-4322-b246-5a439d4c6b62\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j7997" Jul 07 15:41:32.628043 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.628023 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8518e838-f15a-4322-b246-5a439d4c6b62-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-j7997\" (UID: \"8518e838-f15a-4322-b246-5a439d4c6b62\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j7997" Jul 07 15:41:32.716800 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.716755 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j7997" Jul 07 15:41:32.876595 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.876301 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-vfwk5" event={"ID":"a278ee42-38b2-4aee-b359-602e3bbf59be","Type":"ContainerStarted","Data":"f0d527b8b1118aaf568cd78ad6214746f48121a17488fdda9f981bc657ec1e91"} Jul 07 15:41:32.878174 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.878095 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c996ddb-swsbz" event={"ID":"a7338a0a-ea21-4f23-ae05-625d9b884463","Type":"ContainerStarted","Data":"ec3ceddbd2ef2fa2b389a68b1062eb8ef25d325949e61903bcbcbff1dd3a4bdc"} Jul 07 15:41:32.880827 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.880790 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q4x4h" event={"ID":"0eb10328-3df9-435e-91e4-02e0a6731581","Type":"ContainerStarted","Data":"17277250bd63936ed8d98bf54059c07311803715035756ec486dff17a07ca036"} Jul 07 15:41:32.882890 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.882862 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j" event={"ID":"cdd40d24-e30b-496c-b92b-7c80e6ee0b33","Type":"ContainerStarted","Data":"1cf2fb7961d4491f7eab208d336721eb1db70a749f7f6d4138fb1e20f93e74b6"} Jul 07 15:41:32.885391 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.885354 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q2m8b" event={"ID":"b99c24f7-0333-47fe-8b1a-ed9c6a10d23c","Type":"ContainerStarted","Data":"916fde39d5cee7811b554d124eaed6fd9c33db232acf1a1546fcfeffee40f4f3"} Jul 07 15:41:32.887985 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.887962 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j7997"] Jul 07 15:41:32.889409 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.888805 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jrjtw" event={"ID":"8f635da8-3f7f-4450-bdc5-bb8616c2fb12","Type":"ContainerStarted","Data":"d642088415635cd7b977a6414bf6f39260bfd6bb1cbf91279c99a48e268f3162"} Jul 07 15:41:32.889409 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.888860 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jrjtw" event={"ID":"8f635da8-3f7f-4450-bdc5-bb8616c2fb12","Type":"ContainerStarted","Data":"1a36242fc72e45a620cd6b87aec8803baad0733bd1841bcc880b77046d46addc"} Jul 07 15:41:32.890917 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.890887 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-6zs7q" event={"ID":"e26092ce-3f22-4a9e-b816-428098ee2569","Type":"ContainerStarted","Data":"e4bb11c87d3878ce29bba6c33fbf37aeff56fc64225531f4dbbd474378dea568"} Jul 07 15:41:32.892338 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.892058 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-vfwk5" podStartSLOduration=35.359213114 podStartE2EDuration="41.89204074s" totalImagesPullingTime="6.532827626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:32.891993583 +0000 UTC m=+77.069638057" watchObservedRunningTime="2026-07-07 15:41:32.89204074 +0000 UTC m=+77.069685193" Jul 07 15:41:32.896214 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.896190 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-spwqp" event={"ID":"4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3","Type":"ContainerStarted","Data":"7cd2af71111e462d5d47e434483de373c710a58a309d32b819428fc37f56f99b"} Jul 07 15:41:32.897336 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.897196 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-spwqp" event={"ID":"4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3","Type":"ContainerStarted","Data":"817a6a4062defd08e844723e662f5c2491b036e8ea15b32e27a8786d93dd55aa"} Jul 07 15:41:32.897336 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.897240 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-spwqp" Jul 07 15:41:32.911361 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.910022 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8mf6j" podStartSLOduration=65.853118718 podStartE2EDuration="1m12.910000391s" totalImagesPullingTime="7.056881673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:32.909186418 +0000 UTC m=+77.086830893" watchObservedRunningTime="2026-07-07 15:41:32.910000391 +0000 UTC m=+77.087644842" Jul 07 15:41:32.927254 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.927201 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-q2m8b" podStartSLOduration=67.513461801 podStartE2EDuration="1m15.927181158s" totalImagesPullingTime="8.413719357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:32.925840094 +0000 UTC m=+77.103484569" watchObservedRunningTime="2026-07-07 15:41:32.927181158 +0000 UTC m=+77.104825586" Jul 07 15:41:32.948893 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.948657 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-jrjtw" podStartSLOduration=65.878353735 podStartE2EDuration="1m12.948634304s" totalImagesPullingTime="7.070280569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:32.946259167 +0000 UTC m=+77.123903624" watchObservedRunningTime="2026-07-07 15:41:32.948634304 +0000 UTC m=+77.126278760" Jul 07 15:41:32.967363 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:32.967295 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-q4x4h" podStartSLOduration=17.932481709 podStartE2EDuration="29.967274828s" totalImagesPullingTime="12.034793119s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:32.965979411 +0000 UTC m=+77.143623862" watchObservedRunningTime="2026-07-07 15:41:32.967274828 +0000 UTC m=+77.144919278" Jul 07 15:41:33.906045 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:33.904227 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j7997" event={"ID":"8518e838-f15a-4322-b246-5a439d4c6b62","Type":"ContainerStarted","Data":"9e2a95980630ade7c64a49e2377d12e6e7f0009ba6c2f0d698f0152af2419ac0"} Jul 07 15:41:36.920683 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:36.920643 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c996ddb-swsbz" event={"ID":"a7338a0a-ea21-4f23-ae05-625d9b884463","Type":"ContainerStarted","Data":"b827d3b78d4100b6e59f28449937ce579c58dbeb7a7ac82249a202dd35cb420e"} Jul 07 15:41:36.922562 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:36.922534 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j7997" event={"ID":"8518e838-f15a-4322-b246-5a439d4c6b62","Type":"ContainerStarted","Data":"9ff2c37af5f8a8cf002afabb6c1f4aca348a1219864fc9fc7d1d8bdfca6c13e9"} Jul 07 15:41:36.922803 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:36.922782 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j7997" Jul 07 15:41:36.928186 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:36.928160 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j7997" Jul 07 15:41:36.940511 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:36.940456 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74c996ddb-swsbz" podStartSLOduration=2.827001046 podStartE2EDuration="6.940435204s" totalImagesPullingTime="4.113434158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:36.939423613 +0000 UTC m=+81.117068062" watchObservedRunningTime="2026-07-07 15:41:36.940435204 +0000 UTC m=+81.118079655" Jul 07 15:41:36.941516 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:36.941467 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-spwqp" podStartSLOduration=37.85548827 podStartE2EDuration="44.941456113s" totalImagesPullingTime="7.085967843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:32.988900323 +0000 UTC m=+77.166544777" watchObservedRunningTime="2026-07-07 15:41:36.941456113 +0000 UTC m=+81.119100563" Jul 07 15:41:36.958362 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:36.958301 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j7997" podStartSLOduration=1.793807069 podStartE2EDuration="4.95828292s" totalImagesPullingTime="3.164475851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:36.957818919 +0000 UTC m=+81.135463370" watchObservedRunningTime="2026-07-07 15:41:36.95828292 +0000 UTC m=+81.135927371" Jul 07 15:41:40.887884 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:40.887740 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:40.887884 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:40.887819 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:40.893529 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:40.893503 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:40.944400 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:40.944375 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:41:42.907942 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:42.907909 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-spwqp" Jul 07 15:41:43.891504 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:43.891448 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vvrpg"] Jul 07 15:41:43.894283 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:43.894259 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:43.896427 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:43.896395 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 15:41:43.896568 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:43.896480 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zws6j\"" Jul 07 15:41:43.896568 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:43.896408 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 15:41:43.896956 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:43.896919 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 15:41:43.897224 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:43.897153 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 15:41:43.933042 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:43.932423 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c089f7e-a5c2-4343-8074-d016179669ae-node-exporter-wtmp\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:43.933042 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:43.932692 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c089f7e-a5c2-4343-8074-d016179669ae-node-exporter-tls\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:43.933042 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:43.932726 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c089f7e-a5c2-4343-8074-d016179669ae-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:43.933042 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:43.932750 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c089f7e-a5c2-4343-8074-d016179669ae-metrics-client-ca\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:43.933042 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:43.932784 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c089f7e-a5c2-4343-8074-d016179669ae-sys\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:43.933042 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:43.932810 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c089f7e-a5c2-4343-8074-d016179669ae-node-exporter-textfile\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:43.933042 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:43.932843 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c089f7e-a5c2-4343-8074-d016179669ae-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:43.933042 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:43.932887 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c089f7e-a5c2-4343-8074-d016179669ae-root\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:43.933042 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:43.932930 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57zcj\" (UniqueName: \"kubernetes.io/projected/5c089f7e-a5c2-4343-8074-d016179669ae-kube-api-access-57zcj\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:44.033578 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:44.033546 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c089f7e-a5c2-4343-8074-d016179669ae-sys\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:44.033578 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:44.033580 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c089f7e-a5c2-4343-8074-d016179669ae-node-exporter-textfile\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:44.033850 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:44.033616 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c089f7e-a5c2-4343-8074-d016179669ae-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:44.033850 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:44.033660 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c089f7e-a5c2-4343-8074-d016179669ae-root\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:44.033850 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:44.033699 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57zcj\" (UniqueName: \"kubernetes.io/projected/5c089f7e-a5c2-4343-8074-d016179669ae-kube-api-access-57zcj\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:44.033850 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:44.033730 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c089f7e-a5c2-4343-8074-d016179669ae-sys\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:44.034068 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:44.033902 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c089f7e-a5c2-4343-8074-d016179669ae-node-exporter-wtmp\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:44.034068 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:44.033782 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c089f7e-a5c2-4343-8074-d016179669ae-node-exporter-wtmp\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:44.034068 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:44.034018 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c089f7e-a5c2-4343-8074-d016179669ae-node-exporter-tls\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:44.034068 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:44.034038 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c089f7e-a5c2-4343-8074-d016179669ae-node-exporter-textfile\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:44.034068 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:44.034058 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c089f7e-a5c2-4343-8074-d016179669ae-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:44.034354 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:44.034084 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c089f7e-a5c2-4343-8074-d016179669ae-metrics-client-ca\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:44.034354 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:44.034080 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c089f7e-a5c2-4343-8074-d016179669ae-root\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:44.034644 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:44.034597 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c089f7e-a5c2-4343-8074-d016179669ae-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:44.035023 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:44.035001 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c089f7e-a5c2-4343-8074-d016179669ae-metrics-client-ca\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:44.036750 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:44.036723 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c089f7e-a5c2-4343-8074-d016179669ae-node-exporter-tls\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:44.036923 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:44.036905 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c089f7e-a5c2-4343-8074-d016179669ae-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:44.043538 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:44.043512 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57zcj\" (UniqueName: \"kubernetes.io/projected/5c089f7e-a5c2-4343-8074-d016179669ae-kube-api-access-57zcj\") pod \"node-exporter-vvrpg\" (UID: \"5c089f7e-a5c2-4343-8074-d016179669ae\") " pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:44.206341 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:44.206259 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vvrpg" Jul 07 15:41:45.841792 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:45.841750 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:41:49.181491 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:49.181458 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-848545c49c-thmhm"] Jul 07 15:41:50.879623 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:41:50.879582 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c089f7e_a5c2_4343_8074_d016179669ae.slice/crio-6e7deb074cfbd2b71b744d572d35cfa0aec42eae368ae91f8072680a98bc3a1b WatchSource:0}: Error finding container 6e7deb074cfbd2b71b744d572d35cfa0aec42eae368ae91f8072680a98bc3a1b: Status 404 returned error can't find the container with id 6e7deb074cfbd2b71b744d572d35cfa0aec42eae368ae91f8072680a98bc3a1b Jul 07 15:41:50.984961 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:50.984911 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvrpg" event={"ID":"5c089f7e-a5c2-4343-8074-d016179669ae","Type":"ContainerStarted","Data":"6e7deb074cfbd2b71b744d572d35cfa0aec42eae368ae91f8072680a98bc3a1b"} Jul 07 15:41:51.991683 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:51.991648 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-6zs7q" event={"ID":"e26092ce-3f22-4a9e-b816-428098ee2569","Type":"ContainerStarted","Data":"7bdc0cdcff2bf9da37cc044c6c98ef525eb18061b5a6d5c5fadeb1d05bde0845"} Jul 07 15:41:51.992135 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:51.991712 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-6zs7q" Jul 07 15:41:51.997544 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:51.997519 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-6zs7q" Jul 07 15:41:52.010157 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:52.010076 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-6zs7q" podStartSLOduration=5.727751372 podStartE2EDuration="25.01005709s" totalImagesPullingTime="19.282305718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:52.009100892 +0000 UTC m=+96.186745364" watchObservedRunningTime="2026-07-07 15:41:52.01005709 +0000 UTC m=+96.187701542" Jul 07 15:41:53.000520 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:53.000473 2557 generic.go:356] "Generic (PLEG): container finished" podID="5c089f7e-a5c2-4343-8074-d016179669ae" containerID="e940e635bede458c27428a424daa5d3ce6e443b9b6950ca4fc6a28615a4218d9" exitCode=0 Jul 07 15:41:53.001021 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:53.000570 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvrpg" event={"ID":"5c089f7e-a5c2-4343-8074-d016179669ae","Type":"ContainerDied","Data":"e940e635bede458c27428a424daa5d3ce6e443b9b6950ca4fc6a28615a4218d9"} Jul 07 15:41:54.008924 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:54.008886 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvrpg" event={"ID":"5c089f7e-a5c2-4343-8074-d016179669ae","Type":"ContainerStarted","Data":"7fb43f858812e1bcb7e872995d0014ecfc94bd33ee88eec503eb7760fd5d2c31"} Jul 07 15:41:54.009423 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:54.008952 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvrpg" event={"ID":"5c089f7e-a5c2-4343-8074-d016179669ae","Type":"ContainerStarted","Data":"c4f6130f4b1d1f1fe7875525206d70a698b7cbbf57368de363ab8c4009551afd"} Jul 07 15:41:54.029308 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:54.029250 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vvrpg" podStartSLOduration=9.897881774 podStartE2EDuration="11.02922896s" totalImagesPullingTime="1.021119341s" totalInitContainerRuntime="110.227845ms" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:54.027312606 +0000 UTC m=+98.204957058" watchObservedRunningTime="2026-07-07 15:41:54.02922896 +0000 UTC m=+98.206873413" Jul 07 15:41:54.827363 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:41:54.827327 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hwwk7" Jul 07 15:42:08.344009 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:08.343968 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74c996ddb-swsbz"] Jul 07 15:42:13.090937 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:13.090900 2557 generic.go:356] "Generic (PLEG): container finished" podID="37e8dea9-9c9a-41f5-bd64-6d0331ced375" containerID="9b4af3ed0b3ddcc95f318762013974eb6f05ae46a4d599fa23e7b7f49110f8e1" exitCode=0 Jul 07 15:42:13.091382 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:13.090950 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" event={"ID":"37e8dea9-9c9a-41f5-bd64-6d0331ced375","Type":"ContainerDied","Data":"9b4af3ed0b3ddcc95f318762013974eb6f05ae46a4d599fa23e7b7f49110f8e1"} Jul 07 15:42:13.091382 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:13.091359 2557 scope.go:117] "RemoveContainer" containerID="9b4af3ed0b3ddcc95f318762013974eb6f05ae46a4d599fa23e7b7f49110f8e1" Jul 07 15:42:14.096750 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.096718 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-v6x9c" event={"ID":"37e8dea9-9c9a-41f5-bd64-6d0331ced375","Type":"ContainerStarted","Data":"800bc90b195095ff5790e5c7f61904c67956fc39330149da9dfd552801e57790"} Jul 07 15:42:14.210457 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.210416 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-848545c49c-thmhm" podUID="ce7245b7-e2e6-43df-9ae2-568fb739ed49" containerName="registry" containerID="cri-o://b03e154d91193275dcc9be575eeefcfc3a869605e7561f9f5d9c242031f32f1e" gracePeriod=30 Jul 07 15:42:14.485306 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.485275 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:42:14.605694 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.605665 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pnst\" (UniqueName: \"kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-kube-api-access-4pnst\") pod \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " Jul 07 15:42:14.605868 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.605705 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-certificates\") pod \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " Jul 07 15:42:14.605868 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.605729 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-bound-sa-token\") pod \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " Jul 07 15:42:14.605953 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.605928 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce7245b7-e2e6-43df-9ae2-568fb739ed49-image-registry-private-configuration\") pod \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " Jul 07 15:42:14.606032 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.606016 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce7245b7-e2e6-43df-9ae2-568fb739ed49-installation-pull-secrets\") pod \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " Jul 07 15:42:14.606083 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.606068 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce7245b7-e2e6-43df-9ae2-568fb739ed49-ca-trust-extracted\") pod \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " Jul 07 15:42:14.606158 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.606106 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce7245b7-e2e6-43df-9ae2-568fb739ed49-trusted-ca\") pod \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " Jul 07 15:42:14.606216 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.606163 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls\") pod \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\" (UID: \"ce7245b7-e2e6-43df-9ae2-568fb739ed49\") " Jul 07 15:42:14.606216 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.606171 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ce7245b7-e2e6-43df-9ae2-568fb739ed49" (UID: "ce7245b7-e2e6-43df-9ae2-568fb739ed49"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:42:14.606680 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.606411 2557 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-certificates\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:42:14.606680 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.606610 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce7245b7-e2e6-43df-9ae2-568fb739ed49-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ce7245b7-e2e6-43df-9ae2-568fb739ed49" (UID: "ce7245b7-e2e6-43df-9ae2-568fb739ed49"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:42:14.608591 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.608563 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce7245b7-e2e6-43df-9ae2-568fb739ed49-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ce7245b7-e2e6-43df-9ae2-568fb739ed49" (UID: "ce7245b7-e2e6-43df-9ae2-568fb739ed49"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:42:14.608695 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.608603 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ce7245b7-e2e6-43df-9ae2-568fb739ed49" (UID: "ce7245b7-e2e6-43df-9ae2-568fb739ed49"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:42:14.608748 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.608686 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce7245b7-e2e6-43df-9ae2-568fb739ed49-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ce7245b7-e2e6-43df-9ae2-568fb739ed49" (UID: "ce7245b7-e2e6-43df-9ae2-568fb739ed49"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:42:14.608748 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.608730 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-kube-api-access-4pnst" (OuterVolumeSpecName: "kube-api-access-4pnst") pod "ce7245b7-e2e6-43df-9ae2-568fb739ed49" (UID: "ce7245b7-e2e6-43df-9ae2-568fb739ed49"). InnerVolumeSpecName "kube-api-access-4pnst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:42:14.608816 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.608772 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ce7245b7-e2e6-43df-9ae2-568fb739ed49" (UID: "ce7245b7-e2e6-43df-9ae2-568fb739ed49"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:42:14.614458 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.614430 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce7245b7-e2e6-43df-9ae2-568fb739ed49-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ce7245b7-e2e6-43df-9ae2-568fb739ed49" (UID: "ce7245b7-e2e6-43df-9ae2-568fb739ed49"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:42:14.707576 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.707492 2557 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce7245b7-e2e6-43df-9ae2-568fb739ed49-ca-trust-extracted\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:42:14.707576 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.707520 2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce7245b7-e2e6-43df-9ae2-568fb739ed49-trusted-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:42:14.707576 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.707529 2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-registry-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:42:14.707576 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.707538 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pnst\" (UniqueName: \"kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-kube-api-access-4pnst\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:42:14.707576 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.707546 2557 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce7245b7-e2e6-43df-9ae2-568fb739ed49-bound-sa-token\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:42:14.707576 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.707556 2557 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce7245b7-e2e6-43df-9ae2-568fb739ed49-image-registry-private-configuration\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:42:14.707576 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:14.707565 2557 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce7245b7-e2e6-43df-9ae2-568fb739ed49-installation-pull-secrets\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:42:15.101831 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:15.101791 2557 generic.go:356] "Generic (PLEG): container finished" podID="ce7245b7-e2e6-43df-9ae2-568fb739ed49" containerID="b03e154d91193275dcc9be575eeefcfc3a869605e7561f9f5d9c242031f32f1e" exitCode=0 Jul 07 15:42:15.102345 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:15.101853 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-848545c49c-thmhm" Jul 07 15:42:15.102345 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:15.101868 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-848545c49c-thmhm" event={"ID":"ce7245b7-e2e6-43df-9ae2-568fb739ed49","Type":"ContainerDied","Data":"b03e154d91193275dcc9be575eeefcfc3a869605e7561f9f5d9c242031f32f1e"} Jul 07 15:42:15.102345 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:15.101942 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-848545c49c-thmhm" event={"ID":"ce7245b7-e2e6-43df-9ae2-568fb739ed49","Type":"ContainerDied","Data":"dd2fec7ada1f7dcb777f39738a572ee350d7c8dffb5d89faf92112747ce4049c"} Jul 07 15:42:15.102345 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:15.101962 2557 scope.go:117] "RemoveContainer" containerID="b03e154d91193275dcc9be575eeefcfc3a869605e7561f9f5d9c242031f32f1e" Jul 07 15:42:15.110761 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:15.110739 2557 scope.go:117] "RemoveContainer" containerID="b03e154d91193275dcc9be575eeefcfc3a869605e7561f9f5d9c242031f32f1e" Jul 07 15:42:15.111035 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:42:15.111015 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03e154d91193275dcc9be575eeefcfc3a869605e7561f9f5d9c242031f32f1e\": container with ID starting with b03e154d91193275dcc9be575eeefcfc3a869605e7561f9f5d9c242031f32f1e not found: ID does not exist" containerID="b03e154d91193275dcc9be575eeefcfc3a869605e7561f9f5d9c242031f32f1e" Jul 07 15:42:15.111102 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:15.111047 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03e154d91193275dcc9be575eeefcfc3a869605e7561f9f5d9c242031f32f1e"} err="failed to get container status \"b03e154d91193275dcc9be575eeefcfc3a869605e7561f9f5d9c242031f32f1e\": rpc error: code = NotFound desc = could not find container \"b03e154d91193275dcc9be575eeefcfc3a869605e7561f9f5d9c242031f32f1e\": container with ID starting with b03e154d91193275dcc9be575eeefcfc3a869605e7561f9f5d9c242031f32f1e not found: ID does not exist" Jul 07 15:42:15.122123 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:15.122092 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-848545c49c-thmhm"] Jul 07 15:42:15.126127 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:15.126093 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-848545c49c-thmhm"] Jul 07 15:42:16.350513 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:16.350472 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce7245b7-e2e6-43df-9ae2-568fb739ed49" path="/var/lib/kubelet/pods/ce7245b7-e2e6-43df-9ae2-568fb739ed49/volumes" Jul 07 15:42:28.157214 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:28.157174 2557 generic.go:356] "Generic (PLEG): container finished" podID="acc35711-b1b5-4807-9228-70b97021ffb6" containerID="c50f53bc41ce004f746c6ee69d8a820546dcc410ff87bd48a1a829b4949b362b" exitCode=0 Jul 07 15:42:28.157730 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:28.157257 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8tdvw" event={"ID":"acc35711-b1b5-4807-9228-70b97021ffb6","Type":"ContainerDied","Data":"c50f53bc41ce004f746c6ee69d8a820546dcc410ff87bd48a1a829b4949b362b"} Jul 07 15:42:28.157730 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:28.157582 2557 scope.go:117] "RemoveContainer" containerID="c50f53bc41ce004f746c6ee69d8a820546dcc410ff87bd48a1a829b4949b362b" Jul 07 15:42:29.165643 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:29.165610 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8tdvw" event={"ID":"acc35711-b1b5-4807-9228-70b97021ffb6","Type":"ContainerStarted","Data":"520f68bcce54a36087fa7750cc927c4f64a76f7edd3a609ca37657b092c065ac"} Jul 07 15:42:33.366189 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:33.366143 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-74c996ddb-swsbz" podUID="a7338a0a-ea21-4f23-ae05-625d9b884463" containerName="console" containerID="cri-o://b827d3b78d4100b6e59f28449937ce579c58dbeb7a7ac82249a202dd35cb420e" gracePeriod=15 Jul 07 15:42:33.644587 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:33.644502 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:42:33.763234 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:33.763192 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7338a0a-ea21-4f23-ae05-625d9b884463-console-serving-cert\") pod \"a7338a0a-ea21-4f23-ae05-625d9b884463\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " Jul 07 15:42:33.763422 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:33.763260 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7338a0a-ea21-4f23-ae05-625d9b884463-oauth-serving-cert\") pod \"a7338a0a-ea21-4f23-ae05-625d9b884463\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " Jul 07 15:42:33.763422 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:33.763290 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7338a0a-ea21-4f23-ae05-625d9b884463-service-ca\") pod \"a7338a0a-ea21-4f23-ae05-625d9b884463\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " Jul 07 15:42:33.763422 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:33.763340 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7338a0a-ea21-4f23-ae05-625d9b884463-console-config\") pod \"a7338a0a-ea21-4f23-ae05-625d9b884463\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " Jul 07 15:42:33.763422 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:33.763365 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7338a0a-ea21-4f23-ae05-625d9b884463-console-oauth-config\") pod \"a7338a0a-ea21-4f23-ae05-625d9b884463\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " Jul 07 15:42:33.763422 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:33.763397 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpj4h\" (UniqueName: \"kubernetes.io/projected/a7338a0a-ea21-4f23-ae05-625d9b884463-kube-api-access-qpj4h\") pod \"a7338a0a-ea21-4f23-ae05-625d9b884463\" (UID: \"a7338a0a-ea21-4f23-ae05-625d9b884463\") " Jul 07 15:42:33.763782 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:33.763746 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7338a0a-ea21-4f23-ae05-625d9b884463-service-ca" (OuterVolumeSpecName: "service-ca") pod "a7338a0a-ea21-4f23-ae05-625d9b884463" (UID: "a7338a0a-ea21-4f23-ae05-625d9b884463"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:42:33.763865 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:33.763761 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7338a0a-ea21-4f23-ae05-625d9b884463-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a7338a0a-ea21-4f23-ae05-625d9b884463" (UID: "a7338a0a-ea21-4f23-ae05-625d9b884463"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:42:33.763865 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:33.763817 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7338a0a-ea21-4f23-ae05-625d9b884463-console-config" (OuterVolumeSpecName: "console-config") pod "a7338a0a-ea21-4f23-ae05-625d9b884463" (UID: "a7338a0a-ea21-4f23-ae05-625d9b884463"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:42:33.765667 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:33.765632 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7338a0a-ea21-4f23-ae05-625d9b884463-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a7338a0a-ea21-4f23-ae05-625d9b884463" (UID: "a7338a0a-ea21-4f23-ae05-625d9b884463"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:42:33.765782 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:33.765757 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7338a0a-ea21-4f23-ae05-625d9b884463-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a7338a0a-ea21-4f23-ae05-625d9b884463" (UID: "a7338a0a-ea21-4f23-ae05-625d9b884463"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:42:33.765823 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:33.765773 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7338a0a-ea21-4f23-ae05-625d9b884463-kube-api-access-qpj4h" (OuterVolumeSpecName: "kube-api-access-qpj4h") pod "a7338a0a-ea21-4f23-ae05-625d9b884463" (UID: "a7338a0a-ea21-4f23-ae05-625d9b884463"). InnerVolumeSpecName "kube-api-access-qpj4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:42:33.864280 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:33.864242 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7338a0a-ea21-4f23-ae05-625d9b884463-oauth-serving-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:42:33.864280 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:33.864274 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7338a0a-ea21-4f23-ae05-625d9b884463-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:42:33.864280 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:33.864284 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7338a0a-ea21-4f23-ae05-625d9b884463-console-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:42:33.864509 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:33.864293 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7338a0a-ea21-4f23-ae05-625d9b884463-console-oauth-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:42:33.864509 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:33.864301 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpj4h\" (UniqueName: \"kubernetes.io/projected/a7338a0a-ea21-4f23-ae05-625d9b884463-kube-api-access-qpj4h\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:42:33.864509 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:33.864310 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7338a0a-ea21-4f23-ae05-625d9b884463-console-serving-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:42:34.187186 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:34.187154 2557 generic.go:356] "Generic (PLEG): container finished" podID="a7338a0a-ea21-4f23-ae05-625d9b884463" containerID="b827d3b78d4100b6e59f28449937ce579c58dbeb7a7ac82249a202dd35cb420e" exitCode=0 Jul 07 15:42:34.187365 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:34.187199 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c996ddb-swsbz" Jul 07 15:42:34.187365 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:34.187243 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c996ddb-swsbz" event={"ID":"a7338a0a-ea21-4f23-ae05-625d9b884463","Type":"ContainerDied","Data":"b827d3b78d4100b6e59f28449937ce579c58dbeb7a7ac82249a202dd35cb420e"} Jul 07 15:42:34.187365 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:34.187289 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c996ddb-swsbz" event={"ID":"a7338a0a-ea21-4f23-ae05-625d9b884463","Type":"ContainerDied","Data":"ec3ceddbd2ef2fa2b389a68b1062eb8ef25d325949e61903bcbcbff1dd3a4bdc"} Jul 07 15:42:34.187365 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:34.187308 2557 scope.go:117] "RemoveContainer" containerID="b827d3b78d4100b6e59f28449937ce579c58dbeb7a7ac82249a202dd35cb420e" Jul 07 15:42:34.196197 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:34.196182 2557 scope.go:117] "RemoveContainer" containerID="b827d3b78d4100b6e59f28449937ce579c58dbeb7a7ac82249a202dd35cb420e" Jul 07 15:42:34.196464 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:42:34.196446 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b827d3b78d4100b6e59f28449937ce579c58dbeb7a7ac82249a202dd35cb420e\": container with ID starting with b827d3b78d4100b6e59f28449937ce579c58dbeb7a7ac82249a202dd35cb420e not found: ID does not exist" containerID="b827d3b78d4100b6e59f28449937ce579c58dbeb7a7ac82249a202dd35cb420e" Jul 07 15:42:34.196517 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:34.196475 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b827d3b78d4100b6e59f28449937ce579c58dbeb7a7ac82249a202dd35cb420e"} err="failed to get container status \"b827d3b78d4100b6e59f28449937ce579c58dbeb7a7ac82249a202dd35cb420e\": rpc error: code = NotFound desc = could not find container \"b827d3b78d4100b6e59f28449937ce579c58dbeb7a7ac82249a202dd35cb420e\": container with ID starting with b827d3b78d4100b6e59f28449937ce579c58dbeb7a7ac82249a202dd35cb420e not found: ID does not exist" Jul 07 15:42:34.206871 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:34.206850 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74c996ddb-swsbz"] Jul 07 15:42:34.210447 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:34.210426 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74c996ddb-swsbz"] Jul 07 15:42:34.350240 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:34.350201 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7338a0a-ea21-4f23-ae05-625d9b884463" path="/var/lib/kubelet/pods/a7338a0a-ea21-4f23-ae05-625d9b884463/volumes" Jul 07 15:42:38.209617 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:38.209525 2557 generic.go:356] "Generic (PLEG): container finished" podID="62417042-0998-4d6a-ba6a-71ec72bde8e6" containerID="1c51fb596023aebaf20d442b0b0bde9900a2380b02ec9df7798acc05e0312d9b" exitCode=0 Jul 07 15:42:38.209617 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:38.209585 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kt6w8" event={"ID":"62417042-0998-4d6a-ba6a-71ec72bde8e6","Type":"ContainerDied","Data":"1c51fb596023aebaf20d442b0b0bde9900a2380b02ec9df7798acc05e0312d9b"} Jul 07 15:42:38.210147 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:38.209983 2557 scope.go:117] "RemoveContainer" containerID="1c51fb596023aebaf20d442b0b0bde9900a2380b02ec9df7798acc05e0312d9b" Jul 07 15:42:39.215597 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:42:39.215553 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kt6w8" event={"ID":"62417042-0998-4d6a-ba6a-71ec72bde8e6","Type":"ContainerStarted","Data":"3e70834becd5d00153d5e67c73be3c33bbd507d6fc83af961bc06d954026098c"} Jul 07 15:43:13.452921 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.452841 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-787c787b88-5t5fn"] Jul 07 15:43:13.453387 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.453182 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7338a0a-ea21-4f23-ae05-625d9b884463" containerName="console" Jul 07 15:43:13.453387 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.453198 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7338a0a-ea21-4f23-ae05-625d9b884463" containerName="console" Jul 07 15:43:13.453387 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.453214 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce7245b7-e2e6-43df-9ae2-568fb739ed49" containerName="registry" Jul 07 15:43:13.453387 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.453219 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce7245b7-e2e6-43df-9ae2-568fb739ed49" containerName="registry" Jul 07 15:43:13.453387 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.453269 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7338a0a-ea21-4f23-ae05-625d9b884463" containerName="console" Jul 07 15:43:13.453387 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.453280 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce7245b7-e2e6-43df-9ae2-568fb739ed49" containerName="registry" Jul 07 15:43:13.455060 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.455039 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.457012 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.456990 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 15:43:13.458158 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.458102 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-t87sb\"" Jul 07 15:43:13.458776 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.458755 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 15:43:13.458882 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.458860 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 15:43:13.459045 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.459026 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 15:43:13.459387 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.459369 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 15:43:13.464834 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.464814 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 15:43:13.469279 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.469257 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-787c787b88-5t5fn"] Jul 07 15:43:13.482878 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.482851 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc2cx\" (UniqueName: \"kubernetes.io/projected/8419d369-c539-4f2e-9eb2-caaaf639d23b-kube-api-access-lc2cx\") pod \"console-787c787b88-5t5fn\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.482981 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.482909 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8419d369-c539-4f2e-9eb2-caaaf639d23b-console-serving-cert\") pod \"console-787c787b88-5t5fn\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.482981 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.482936 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-service-ca\") pod \"console-787c787b88-5t5fn\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.483099 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.483001 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-oauth-serving-cert\") pod \"console-787c787b88-5t5fn\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.483099 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.483078 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-trusted-ca-bundle\") pod \"console-787c787b88-5t5fn\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.483228 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.483129 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8419d369-c539-4f2e-9eb2-caaaf639d23b-console-oauth-config\") pod \"console-787c787b88-5t5fn\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.483228 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.483155 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-console-config\") pod \"console-787c787b88-5t5fn\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.583972 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.583928 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc2cx\" (UniqueName: \"kubernetes.io/projected/8419d369-c539-4f2e-9eb2-caaaf639d23b-kube-api-access-lc2cx\") pod \"console-787c787b88-5t5fn\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.584208 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.583993 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8419d369-c539-4f2e-9eb2-caaaf639d23b-console-serving-cert\") pod \"console-787c787b88-5t5fn\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.584208 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.584017 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-service-ca\") pod \"console-787c787b88-5t5fn\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.584208 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.584040 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-oauth-serving-cert\") pod \"console-787c787b88-5t5fn\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.584208 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.584071 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-trusted-ca-bundle\") pod \"console-787c787b88-5t5fn\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.584208 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.584101 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8419d369-c539-4f2e-9eb2-caaaf639d23b-console-oauth-config\") pod \"console-787c787b88-5t5fn\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.584208 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.584138 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-console-config\") pod \"console-787c787b88-5t5fn\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.584864 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.584843 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-console-config\") pod \"console-787c787b88-5t5fn\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.584992 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.584951 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-oauth-serving-cert\") pod \"console-787c787b88-5t5fn\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.585044 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.585023 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-service-ca\") pod \"console-787c787b88-5t5fn\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.585321 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.585302 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-trusted-ca-bundle\") pod \"console-787c787b88-5t5fn\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.586724 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.586698 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8419d369-c539-4f2e-9eb2-caaaf639d23b-console-serving-cert\") pod \"console-787c787b88-5t5fn\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.586825 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.586726 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8419d369-c539-4f2e-9eb2-caaaf639d23b-console-oauth-config\") pod \"console-787c787b88-5t5fn\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.590795 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.590773 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc2cx\" (UniqueName: \"kubernetes.io/projected/8419d369-c539-4f2e-9eb2-caaaf639d23b-kube-api-access-lc2cx\") pod \"console-787c787b88-5t5fn\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.766427 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.766334 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:13.898284 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:13.898257 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-787c787b88-5t5fn"] Jul 07 15:43:13.900995 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:43:13.900952 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8419d369_c539_4f2e_9eb2_caaaf639d23b.slice/crio-cc90d194dfca9106a32e90fd19ff01a998dec908c8c312d7283886fb0c0f29de WatchSource:0}: Error finding container cc90d194dfca9106a32e90fd19ff01a998dec908c8c312d7283886fb0c0f29de: Status 404 returned error can't find the container with id cc90d194dfca9106a32e90fd19ff01a998dec908c8c312d7283886fb0c0f29de Jul 07 15:43:14.350162 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:14.350106 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-787c787b88-5t5fn" event={"ID":"8419d369-c539-4f2e-9eb2-caaaf639d23b","Type":"ContainerStarted","Data":"520ea0496d0251d5fe58f4d8484862eb7a4a8339ebd2ae1b1f780668f53e6474"} Jul 07 15:43:14.350337 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:14.350172 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-787c787b88-5t5fn" event={"ID":"8419d369-c539-4f2e-9eb2-caaaf639d23b","Type":"ContainerStarted","Data":"cc90d194dfca9106a32e90fd19ff01a998dec908c8c312d7283886fb0c0f29de"} Jul 07 15:43:14.365267 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:14.365209 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-787c787b88-5t5fn" podStartSLOduration=1.3651921759999999 podStartE2EDuration="1.365192176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:43:14.364296706 +0000 UTC m=+178.541941155" watchObservedRunningTime="2026-07-07 15:43:14.365192176 +0000 UTC m=+178.542836626" Jul 07 15:43:23.767229 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:23.767183 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:23.767644 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:23.767258 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:23.772054 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:23.772030 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:24.392040 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:24.392009 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:43:55.738025 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:55.737985 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r"] Jul 07 15:43:55.740310 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:55.740292 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r" Jul 07 15:43:55.742357 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:55.742328 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 15:43:55.742514 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:55.742384 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 15:43:55.742591 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:55.742575 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-s9d6n\"" Jul 07 15:43:55.751852 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:55.751828 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r"] Jul 07 15:43:55.847108 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:55.847073 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wvmz\" (UniqueName: \"kubernetes.io/projected/48ceb2b9-40f0-4c64-91fb-7a33ceca0de2-kube-api-access-9wvmz\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r\" (UID: \"48ceb2b9-40f0-4c64-91fb-7a33ceca0de2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r" Jul 07 15:43:55.847301 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:55.847136 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48ceb2b9-40f0-4c64-91fb-7a33ceca0de2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r\" (UID: \"48ceb2b9-40f0-4c64-91fb-7a33ceca0de2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r" Jul 07 15:43:55.847301 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:55.847200 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48ceb2b9-40f0-4c64-91fb-7a33ceca0de2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r\" (UID: \"48ceb2b9-40f0-4c64-91fb-7a33ceca0de2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r" Jul 07 15:43:55.948454 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:55.948417 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48ceb2b9-40f0-4c64-91fb-7a33ceca0de2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r\" (UID: \"48ceb2b9-40f0-4c64-91fb-7a33ceca0de2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r" Jul 07 15:43:55.948611 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:55.948512 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wvmz\" (UniqueName: \"kubernetes.io/projected/48ceb2b9-40f0-4c64-91fb-7a33ceca0de2-kube-api-access-9wvmz\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r\" (UID: \"48ceb2b9-40f0-4c64-91fb-7a33ceca0de2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r" Jul 07 15:43:55.948611 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:55.948541 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48ceb2b9-40f0-4c64-91fb-7a33ceca0de2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r\" (UID: \"48ceb2b9-40f0-4c64-91fb-7a33ceca0de2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r" Jul 07 15:43:55.948933 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:55.948912 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48ceb2b9-40f0-4c64-91fb-7a33ceca0de2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r\" (UID: \"48ceb2b9-40f0-4c64-91fb-7a33ceca0de2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r" Jul 07 15:43:55.948977 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:55.948912 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48ceb2b9-40f0-4c64-91fb-7a33ceca0de2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r\" (UID: \"48ceb2b9-40f0-4c64-91fb-7a33ceca0de2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r" Jul 07 15:43:55.956714 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:55.956691 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wvmz\" (UniqueName: \"kubernetes.io/projected/48ceb2b9-40f0-4c64-91fb-7a33ceca0de2-kube-api-access-9wvmz\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r\" (UID: \"48ceb2b9-40f0-4c64-91fb-7a33ceca0de2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r" Jul 07 15:43:56.049108 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:56.049077 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r" Jul 07 15:43:56.177136 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:56.177091 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r"] Jul 07 15:43:56.180278 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:43:56.180249 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ceb2b9_40f0_4c64_91fb_7a33ceca0de2.slice/crio-83587f627a17d2900a53829c2d4895d23590b76e3cec6e1ebd072232c3793b21 WatchSource:0}: Error finding container 83587f627a17d2900a53829c2d4895d23590b76e3cec6e1ebd072232c3793b21: Status 404 returned error can't find the container with id 83587f627a17d2900a53829c2d4895d23590b76e3cec6e1ebd072232c3793b21 Jul 07 15:43:56.515312 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:43:56.515230 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r" event={"ID":"48ceb2b9-40f0-4c64-91fb-7a33ceca0de2","Type":"ContainerStarted","Data":"83587f627a17d2900a53829c2d4895d23590b76e3cec6e1ebd072232c3793b21"} Jul 07 15:44:04.547931 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:04.547889 2557 generic.go:356] "Generic (PLEG): container finished" podID="48ceb2b9-40f0-4c64-91fb-7a33ceca0de2" containerID="f8fe7db12f2f8ddddefa636db5d45999d183feeeac5d1a5698ef195eeca95f99" exitCode=0 Jul 07 15:44:04.548449 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:04.547944 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r" event={"ID":"48ceb2b9-40f0-4c64-91fb-7a33ceca0de2","Type":"ContainerDied","Data":"f8fe7db12f2f8ddddefa636db5d45999d183feeeac5d1a5698ef195eeca95f99"} Jul 07 15:44:07.562644 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:07.562609 2557 generic.go:356] "Generic (PLEG): container finished" podID="48ceb2b9-40f0-4c64-91fb-7a33ceca0de2" containerID="1895fdf61885ddb99947fa62508c31823131a5da3ca5bf7b14ce9162fa4ad6a2" exitCode=0 Jul 07 15:44:07.563052 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:07.562692 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r" event={"ID":"48ceb2b9-40f0-4c64-91fb-7a33ceca0de2","Type":"ContainerDied","Data":"1895fdf61885ddb99947fa62508c31823131a5da3ca5bf7b14ce9162fa4ad6a2"} Jul 07 15:44:16.604214 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:16.604176 2557 generic.go:356] "Generic (PLEG): container finished" podID="48ceb2b9-40f0-4c64-91fb-7a33ceca0de2" containerID="91444e2f97ed0a278d535e8c40425815725fb111322087665f1e8dbe11d5bfaf" exitCode=0 Jul 07 15:44:16.604596 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:16.604250 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r" event={"ID":"48ceb2b9-40f0-4c64-91fb-7a33ceca0de2","Type":"ContainerDied","Data":"91444e2f97ed0a278d535e8c40425815725fb111322087665f1e8dbe11d5bfaf"} Jul 07 15:44:17.782853 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:17.782819 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r" Jul 07 15:44:17.823927 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:17.823891 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48ceb2b9-40f0-4c64-91fb-7a33ceca0de2-util\") pod \"48ceb2b9-40f0-4c64-91fb-7a33ceca0de2\" (UID: \"48ceb2b9-40f0-4c64-91fb-7a33ceca0de2\") " Jul 07 15:44:17.824100 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:17.824010 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48ceb2b9-40f0-4c64-91fb-7a33ceca0de2-bundle\") pod \"48ceb2b9-40f0-4c64-91fb-7a33ceca0de2\" (UID: \"48ceb2b9-40f0-4c64-91fb-7a33ceca0de2\") " Jul 07 15:44:17.824100 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:17.824045 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wvmz\" (UniqueName: \"kubernetes.io/projected/48ceb2b9-40f0-4c64-91fb-7a33ceca0de2-kube-api-access-9wvmz\") pod \"48ceb2b9-40f0-4c64-91fb-7a33ceca0de2\" (UID: \"48ceb2b9-40f0-4c64-91fb-7a33ceca0de2\") " Jul 07 15:44:17.824667 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:17.824640 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48ceb2b9-40f0-4c64-91fb-7a33ceca0de2-bundle" (OuterVolumeSpecName: "bundle") pod "48ceb2b9-40f0-4c64-91fb-7a33ceca0de2" (UID: "48ceb2b9-40f0-4c64-91fb-7a33ceca0de2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:44:17.826544 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:17.826510 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ceb2b9-40f0-4c64-91fb-7a33ceca0de2-kube-api-access-9wvmz" (OuterVolumeSpecName: "kube-api-access-9wvmz") pod "48ceb2b9-40f0-4c64-91fb-7a33ceca0de2" (UID: "48ceb2b9-40f0-4c64-91fb-7a33ceca0de2"). InnerVolumeSpecName "kube-api-access-9wvmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:44:17.828000 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:17.827978 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48ceb2b9-40f0-4c64-91fb-7a33ceca0de2-util" (OuterVolumeSpecName: "util") pod "48ceb2b9-40f0-4c64-91fb-7a33ceca0de2" (UID: "48ceb2b9-40f0-4c64-91fb-7a33ceca0de2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:44:17.924921 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:17.924834 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48ceb2b9-40f0-4c64-91fb-7a33ceca0de2-bundle\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:44:17.924921 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:17.924865 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wvmz\" (UniqueName: \"kubernetes.io/projected/48ceb2b9-40f0-4c64-91fb-7a33ceca0de2-kube-api-access-9wvmz\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:44:17.924921 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:17.924875 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48ceb2b9-40f0-4c64-91fb-7a33ceca0de2-util\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:44:18.613296 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:18.613218 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r" Jul 07 15:44:18.613296 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:18.613224 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402482r" event={"ID":"48ceb2b9-40f0-4c64-91fb-7a33ceca0de2","Type":"ContainerDied","Data":"83587f627a17d2900a53829c2d4895d23590b76e3cec6e1ebd072232c3793b21"} Jul 07 15:44:18.613296 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:18.613271 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83587f627a17d2900a53829c2d4895d23590b76e3cec6e1ebd072232c3793b21" Jul 07 15:44:23.851325 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:23.851286 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-42njj"] Jul 07 15:44:23.851804 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:23.851570 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ceb2b9-40f0-4c64-91fb-7a33ceca0de2" containerName="util" Jul 07 15:44:23.851804 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:23.851583 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ceb2b9-40f0-4c64-91fb-7a33ceca0de2" containerName="util" Jul 07 15:44:23.851804 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:23.851596 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ceb2b9-40f0-4c64-91fb-7a33ceca0de2" containerName="pull" Jul 07 15:44:23.851804 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:23.851602 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ceb2b9-40f0-4c64-91fb-7a33ceca0de2" containerName="pull" Jul 07 15:44:23.851804 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:23.851620 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ceb2b9-40f0-4c64-91fb-7a33ceca0de2" containerName="extract" Jul 07 15:44:23.851804 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:23.851626 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ceb2b9-40f0-4c64-91fb-7a33ceca0de2" containerName="extract" Jul 07 15:44:23.851804 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:23.851675 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="48ceb2b9-40f0-4c64-91fb-7a33ceca0de2" containerName="extract" Jul 07 15:44:23.853568 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:23.853551 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-42njj" Jul 07 15:44:23.856483 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:23.856460 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 07 15:44:23.864264 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:23.864248 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-fx4r8\"" Jul 07 15:44:23.864337 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:23.864272 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 07 15:44:23.872700 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:23.872671 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-42njj"] Jul 07 15:44:23.978943 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:23.978909 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b434e12-4121-498e-b831-abf6a0f38ab0-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-42njj\" (UID: \"4b434e12-4121-498e-b831-abf6a0f38ab0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-42njj" Jul 07 15:44:23.979138 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:23.978984 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7zcf\" (UniqueName: \"kubernetes.io/projected/4b434e12-4121-498e-b831-abf6a0f38ab0-kube-api-access-x7zcf\") pod \"cert-manager-operator-controller-manager-55c444bf86-42njj\" (UID: \"4b434e12-4121-498e-b831-abf6a0f38ab0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-42njj" Jul 07 15:44:24.079753 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:24.079708 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7zcf\" (UniqueName: \"kubernetes.io/projected/4b434e12-4121-498e-b831-abf6a0f38ab0-kube-api-access-x7zcf\") pod \"cert-manager-operator-controller-manager-55c444bf86-42njj\" (UID: \"4b434e12-4121-498e-b831-abf6a0f38ab0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-42njj" Jul 07 15:44:24.079753 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:24.079754 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b434e12-4121-498e-b831-abf6a0f38ab0-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-42njj\" (UID: \"4b434e12-4121-498e-b831-abf6a0f38ab0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-42njj" Jul 07 15:44:24.080093 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:24.080076 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b434e12-4121-498e-b831-abf6a0f38ab0-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-42njj\" (UID: \"4b434e12-4121-498e-b831-abf6a0f38ab0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-42njj" Jul 07 15:44:24.088826 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:24.088801 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7zcf\" (UniqueName: \"kubernetes.io/projected/4b434e12-4121-498e-b831-abf6a0f38ab0-kube-api-access-x7zcf\") pod \"cert-manager-operator-controller-manager-55c444bf86-42njj\" (UID: \"4b434e12-4121-498e-b831-abf6a0f38ab0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-42njj" Jul 07 15:44:24.161892 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:24.161787 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-42njj" Jul 07 15:44:24.300234 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:24.300199 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-42njj"] Jul 07 15:44:24.305976 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:44:24.305941 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b434e12_4121_498e_b831_abf6a0f38ab0.slice/crio-726e30766e727d7ca4b7d45cdf3e70bfdbbcdbd3856e24896cebb3569b480ae5 WatchSource:0}: Error finding container 726e30766e727d7ca4b7d45cdf3e70bfdbbcdbd3856e24896cebb3569b480ae5: Status 404 returned error can't find the container with id 726e30766e727d7ca4b7d45cdf3e70bfdbbcdbd3856e24896cebb3569b480ae5 Jul 07 15:44:24.637533 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:24.637495 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-42njj" event={"ID":"4b434e12-4121-498e-b831-abf6a0f38ab0","Type":"ContainerStarted","Data":"726e30766e727d7ca4b7d45cdf3e70bfdbbcdbd3856e24896cebb3569b480ae5"} Jul 07 15:44:26.649595 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:26.649559 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-42njj" event={"ID":"4b434e12-4121-498e-b831-abf6a0f38ab0","Type":"ContainerStarted","Data":"fc43539909f0b4740b26040d2e4c587404df38083356d5a302bccaccedb29cfa"} Jul 07 15:44:26.673996 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:26.673936 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-42njj" podStartSLOduration=2.145637114 podStartE2EDuration="3.673919432s" totalImagesPullingTime="1.528282318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:44:26.671890784 +0000 UTC m=+250.849535234" watchObservedRunningTime="2026-07-07 15:44:26.673919432 +0000 UTC m=+250.851563881" Jul 07 15:44:27.590405 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:27.590368 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff"] Jul 07 15:44:27.592535 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:27.592519 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff" Jul 07 15:44:27.594730 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:27.594685 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-s9d6n\"" Jul 07 15:44:27.595097 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:27.595078 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 15:44:27.595277 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:27.595253 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 15:44:27.607759 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:27.607738 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff"] Jul 07 15:44:27.710386 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:27.710351 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dee01be6-9235-46d9-a811-dd1651beb8ad-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff\" (UID: \"dee01be6-9235-46d9-a811-dd1651beb8ad\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff" Jul 07 15:44:27.710811 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:27.710408 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg6d4\" (UniqueName: \"kubernetes.io/projected/dee01be6-9235-46d9-a811-dd1651beb8ad-kube-api-access-bg6d4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff\" (UID: \"dee01be6-9235-46d9-a811-dd1651beb8ad\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff" Jul 07 15:44:27.710811 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:27.710509 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dee01be6-9235-46d9-a811-dd1651beb8ad-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff\" (UID: \"dee01be6-9235-46d9-a811-dd1651beb8ad\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff" Jul 07 15:44:27.811853 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:27.811814 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dee01be6-9235-46d9-a811-dd1651beb8ad-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff\" (UID: \"dee01be6-9235-46d9-a811-dd1651beb8ad\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff" Jul 07 15:44:27.812026 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:27.811881 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dee01be6-9235-46d9-a811-dd1651beb8ad-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff\" (UID: \"dee01be6-9235-46d9-a811-dd1651beb8ad\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff" Jul 07 15:44:27.812026 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:27.811945 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg6d4\" (UniqueName: \"kubernetes.io/projected/dee01be6-9235-46d9-a811-dd1651beb8ad-kube-api-access-bg6d4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff\" (UID: \"dee01be6-9235-46d9-a811-dd1651beb8ad\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff" Jul 07 15:44:27.812297 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:27.812275 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dee01be6-9235-46d9-a811-dd1651beb8ad-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff\" (UID: \"dee01be6-9235-46d9-a811-dd1651beb8ad\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff" Jul 07 15:44:27.812348 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:27.812330 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dee01be6-9235-46d9-a811-dd1651beb8ad-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff\" (UID: \"dee01be6-9235-46d9-a811-dd1651beb8ad\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff" Jul 07 15:44:27.820394 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:27.820367 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg6d4\" (UniqueName: \"kubernetes.io/projected/dee01be6-9235-46d9-a811-dd1651beb8ad-kube-api-access-bg6d4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff\" (UID: \"dee01be6-9235-46d9-a811-dd1651beb8ad\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff" Jul 07 15:44:27.901966 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:27.901872 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff" Jul 07 15:44:28.041649 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:28.041617 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff"] Jul 07 15:44:28.044576 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:44:28.044549 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee01be6_9235_46d9_a811_dd1651beb8ad.slice/crio-3f08c70797787a426b1cfb021f73a2cbc83a89b0a65fb04f31bffe5e10ab9608 WatchSource:0}: Error finding container 3f08c70797787a426b1cfb021f73a2cbc83a89b0a65fb04f31bffe5e10ab9608: Status 404 returned error can't find the container with id 3f08c70797787a426b1cfb021f73a2cbc83a89b0a65fb04f31bffe5e10ab9608 Jul 07 15:44:28.660257 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:28.660225 2557 generic.go:356] "Generic (PLEG): container finished" podID="dee01be6-9235-46d9-a811-dd1651beb8ad" containerID="129104c6b4f90fd768356b827ad15957e372fcb83ce9f5306ddc9af8b0d31a47" exitCode=0 Jul 07 15:44:28.660448 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:28.660313 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff" event={"ID":"dee01be6-9235-46d9-a811-dd1651beb8ad","Type":"ContainerDied","Data":"129104c6b4f90fd768356b827ad15957e372fcb83ce9f5306ddc9af8b0d31a47"} Jul 07 15:44:28.660448 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:28.660377 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff" event={"ID":"dee01be6-9235-46d9-a811-dd1651beb8ad","Type":"ContainerStarted","Data":"3f08c70797787a426b1cfb021f73a2cbc83a89b0a65fb04f31bffe5e10ab9608"} Jul 07 15:44:31.269636 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:31.269560 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8ztbs"] Jul 07 15:44:31.271505 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:31.271489 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8ztbs" Jul 07 15:44:31.274295 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:31.274271 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-jh5ck\"" Jul 07 15:44:31.274407 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:31.274369 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 07 15:44:31.274480 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:31.274454 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 07 15:44:31.282583 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:31.282562 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8ztbs"] Jul 07 15:44:31.343426 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:31.343390 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b41ca6e-3c54-4e1d-be4d-d1eb867152e2-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8ztbs\" (UID: \"4b41ca6e-3c54-4e1d-be4d-d1eb867152e2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8ztbs" Jul 07 15:44:31.343590 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:31.343440 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56rnw\" (UniqueName: \"kubernetes.io/projected/4b41ca6e-3c54-4e1d-be4d-d1eb867152e2-kube-api-access-56rnw\") pod \"cert-manager-cainjector-559cdd8c87-8ztbs\" (UID: \"4b41ca6e-3c54-4e1d-be4d-d1eb867152e2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8ztbs" Jul 07 15:44:31.444781 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:31.444748 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56rnw\" (UniqueName: \"kubernetes.io/projected/4b41ca6e-3c54-4e1d-be4d-d1eb867152e2-kube-api-access-56rnw\") pod \"cert-manager-cainjector-559cdd8c87-8ztbs\" (UID: \"4b41ca6e-3c54-4e1d-be4d-d1eb867152e2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8ztbs" Jul 07 15:44:31.444954 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:31.444818 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b41ca6e-3c54-4e1d-be4d-d1eb867152e2-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8ztbs\" (UID: \"4b41ca6e-3c54-4e1d-be4d-d1eb867152e2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8ztbs" Jul 07 15:44:31.453101 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:31.453068 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b41ca6e-3c54-4e1d-be4d-d1eb867152e2-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8ztbs\" (UID: \"4b41ca6e-3c54-4e1d-be4d-d1eb867152e2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8ztbs" Jul 07 15:44:31.453250 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:31.453192 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56rnw\" (UniqueName: \"kubernetes.io/projected/4b41ca6e-3c54-4e1d-be4d-d1eb867152e2-kube-api-access-56rnw\") pod \"cert-manager-cainjector-559cdd8c87-8ztbs\" (UID: \"4b41ca6e-3c54-4e1d-be4d-d1eb867152e2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8ztbs" Jul 07 15:44:31.590950 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:31.590920 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8ztbs" Jul 07 15:44:31.678190 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:31.678147 2557 generic.go:356] "Generic (PLEG): container finished" podID="dee01be6-9235-46d9-a811-dd1651beb8ad" containerID="2e6e7982e84d24b152655ebc64e8a1eca80d8bd9d6ad0acb1f414cad68ab282f" exitCode=0 Jul 07 15:44:31.678325 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:31.678287 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff" event={"ID":"dee01be6-9235-46d9-a811-dd1651beb8ad","Type":"ContainerDied","Data":"2e6e7982e84d24b152655ebc64e8a1eca80d8bd9d6ad0acb1f414cad68ab282f"} Jul 07 15:44:31.729073 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:31.729049 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8ztbs"] Jul 07 15:44:31.732198 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:44:31.732177 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b41ca6e_3c54_4e1d_be4d_d1eb867152e2.slice/crio-96a4c99a28b23e14eb451e228d1f82a850ab5a7815d74333e5b544bbe298a597 WatchSource:0}: Error finding container 96a4c99a28b23e14eb451e228d1f82a850ab5a7815d74333e5b544bbe298a597: Status 404 returned error can't find the container with id 96a4c99a28b23e14eb451e228d1f82a850ab5a7815d74333e5b544bbe298a597 Jul 07 15:44:32.687399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:32.687358 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8ztbs" event={"ID":"4b41ca6e-3c54-4e1d-be4d-d1eb867152e2","Type":"ContainerStarted","Data":"96a4c99a28b23e14eb451e228d1f82a850ab5a7815d74333e5b544bbe298a597"} Jul 07 15:44:32.689747 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:32.689715 2557 generic.go:356] "Generic (PLEG): container finished" podID="dee01be6-9235-46d9-a811-dd1651beb8ad" containerID="25780e7c3e1dd15d5d9facbb5c997031c46af4a0812d51067e82873088ae66a5" exitCode=0 Jul 07 15:44:32.689880 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:32.689787 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff" event={"ID":"dee01be6-9235-46d9-a811-dd1651beb8ad","Type":"ContainerDied","Data":"25780e7c3e1dd15d5d9facbb5c997031c46af4a0812d51067e82873088ae66a5"} Jul 07 15:44:34.017372 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:34.017349 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff" Jul 07 15:44:34.169003 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:34.168963 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dee01be6-9235-46d9-a811-dd1651beb8ad-bundle\") pod \"dee01be6-9235-46d9-a811-dd1651beb8ad\" (UID: \"dee01be6-9235-46d9-a811-dd1651beb8ad\") " Jul 07 15:44:34.169183 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:34.169041 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dee01be6-9235-46d9-a811-dd1651beb8ad-util\") pod \"dee01be6-9235-46d9-a811-dd1651beb8ad\" (UID: \"dee01be6-9235-46d9-a811-dd1651beb8ad\") " Jul 07 15:44:34.169183 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:34.169067 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bg6d4\" (UniqueName: \"kubernetes.io/projected/dee01be6-9235-46d9-a811-dd1651beb8ad-kube-api-access-bg6d4\") pod \"dee01be6-9235-46d9-a811-dd1651beb8ad\" (UID: \"dee01be6-9235-46d9-a811-dd1651beb8ad\") " Jul 07 15:44:34.169427 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:34.169402 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee01be6-9235-46d9-a811-dd1651beb8ad-bundle" (OuterVolumeSpecName: "bundle") pod "dee01be6-9235-46d9-a811-dd1651beb8ad" (UID: "dee01be6-9235-46d9-a811-dd1651beb8ad"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:44:34.171681 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:34.171530 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee01be6-9235-46d9-a811-dd1651beb8ad-kube-api-access-bg6d4" (OuterVolumeSpecName: "kube-api-access-bg6d4") pod "dee01be6-9235-46d9-a811-dd1651beb8ad" (UID: "dee01be6-9235-46d9-a811-dd1651beb8ad"). InnerVolumeSpecName "kube-api-access-bg6d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:44:34.173564 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:34.173540 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee01be6-9235-46d9-a811-dd1651beb8ad-util" (OuterVolumeSpecName: "util") pod "dee01be6-9235-46d9-a811-dd1651beb8ad" (UID: "dee01be6-9235-46d9-a811-dd1651beb8ad"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:44:34.269789 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:34.269691 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dee01be6-9235-46d9-a811-dd1651beb8ad-bundle\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:44:34.269789 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:34.269724 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dee01be6-9235-46d9-a811-dd1651beb8ad-util\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:44:34.269789 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:34.269733 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg6d4\" (UniqueName: \"kubernetes.io/projected/dee01be6-9235-46d9-a811-dd1651beb8ad-kube-api-access-bg6d4\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:44:34.701078 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:34.701047 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff" Jul 07 15:44:34.701078 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:34.701067 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdm7ff" event={"ID":"dee01be6-9235-46d9-a811-dd1651beb8ad","Type":"ContainerDied","Data":"3f08c70797787a426b1cfb021f73a2cbc83a89b0a65fb04f31bffe5e10ab9608"} Jul 07 15:44:34.701329 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:34.701139 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f08c70797787a426b1cfb021f73a2cbc83a89b0a65fb04f31bffe5e10ab9608" Jul 07 15:44:34.702579 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:34.702555 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8ztbs" event={"ID":"4b41ca6e-3c54-4e1d-be4d-d1eb867152e2","Type":"ContainerStarted","Data":"44a99407ed86eaae03f223856bcfd13de3240d30a366922fa7ae17e159cd8ce5"} Jul 07 15:44:34.719313 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:34.719271 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8ztbs" podStartSLOduration=1.389278888 podStartE2EDuration="3.719257726s" totalImagesPullingTime="2.329978838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:44:34.718240176 +0000 UTC m=+258.895884620" watchObservedRunningTime="2026-07-07 15:44:34.719257726 +0000 UTC m=+258.896902175" Jul 07 15:44:40.645562 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:40.645478 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnl8"] Jul 07 15:44:40.645919 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:40.645778 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dee01be6-9235-46d9-a811-dd1651beb8ad" containerName="util" Jul 07 15:44:40.645919 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:40.645789 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dee01be6-9235-46d9-a811-dd1651beb8ad" containerName="util" Jul 07 15:44:40.645919 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:40.645807 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dee01be6-9235-46d9-a811-dd1651beb8ad" containerName="extract" Jul 07 15:44:40.645919 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:40.645812 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dee01be6-9235-46d9-a811-dd1651beb8ad" containerName="extract" Jul 07 15:44:40.645919 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:40.645820 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dee01be6-9235-46d9-a811-dd1651beb8ad" containerName="pull" Jul 07 15:44:40.645919 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:40.645825 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dee01be6-9235-46d9-a811-dd1651beb8ad" containerName="pull" Jul 07 15:44:40.645919 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:40.645876 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="dee01be6-9235-46d9-a811-dd1651beb8ad" containerName="extract" Jul 07 15:44:40.648625 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:40.648610 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnl8" Jul 07 15:44:40.650861 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:40.650845 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 15:44:40.651407 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:40.651386 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 15:44:40.651516 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:40.651416 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-zqt2w\"" Jul 07 15:44:40.660366 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:40.660346 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnl8"] Jul 07 15:44:40.722751 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:40.722716 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a554767e-512e-4418-88cf-49b2cf7f3e79-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hwnl8\" (UID: \"a554767e-512e-4418-88cf-49b2cf7f3e79\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnl8" Jul 07 15:44:40.722913 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:40.722754 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz6mr\" (UniqueName: \"kubernetes.io/projected/a554767e-512e-4418-88cf-49b2cf7f3e79-kube-api-access-xz6mr\") pod \"openshift-lws-operator-fd8ccff4c-hwnl8\" (UID: \"a554767e-512e-4418-88cf-49b2cf7f3e79\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnl8" Jul 07 15:44:40.823517 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:40.823487 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a554767e-512e-4418-88cf-49b2cf7f3e79-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hwnl8\" (UID: \"a554767e-512e-4418-88cf-49b2cf7f3e79\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnl8" Jul 07 15:44:40.823681 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:40.823524 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz6mr\" (UniqueName: \"kubernetes.io/projected/a554767e-512e-4418-88cf-49b2cf7f3e79-kube-api-access-xz6mr\") pod \"openshift-lws-operator-fd8ccff4c-hwnl8\" (UID: \"a554767e-512e-4418-88cf-49b2cf7f3e79\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnl8" Jul 07 15:44:40.823889 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:40.823861 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a554767e-512e-4418-88cf-49b2cf7f3e79-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hwnl8\" (UID: \"a554767e-512e-4418-88cf-49b2cf7f3e79\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnl8" Jul 07 15:44:40.832225 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:40.832201 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz6mr\" (UniqueName: \"kubernetes.io/projected/a554767e-512e-4418-88cf-49b2cf7f3e79-kube-api-access-xz6mr\") pod \"openshift-lws-operator-fd8ccff4c-hwnl8\" (UID: \"a554767e-512e-4418-88cf-49b2cf7f3e79\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnl8" Jul 07 15:44:40.957910 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:40.957813 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnl8" Jul 07 15:44:41.106811 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:41.106687 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnl8"] Jul 07 15:44:41.109527 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:44:41.109494 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda554767e_512e_4418_88cf_49b2cf7f3e79.slice/crio-8f78f1fd283e7f1032dfdb73f4eabce65144141a70a2e05e833e8e3452e28798 WatchSource:0}: Error finding container 8f78f1fd283e7f1032dfdb73f4eabce65144141a70a2e05e833e8e3452e28798: Status 404 returned error can't find the container with id 8f78f1fd283e7f1032dfdb73f4eabce65144141a70a2e05e833e8e3452e28798 Jul 07 15:44:41.735217 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:41.735179 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnl8" event={"ID":"a554767e-512e-4418-88cf-49b2cf7f3e79","Type":"ContainerStarted","Data":"8f78f1fd283e7f1032dfdb73f4eabce65144141a70a2e05e833e8e3452e28798"} Jul 07 15:44:45.755370 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:45.755337 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnl8" event={"ID":"a554767e-512e-4418-88cf-49b2cf7f3e79","Type":"ContainerStarted","Data":"01e8cf333513db77f918efd5b7c96da9ce5086abc002c1e5e6084db47089dc34"} Jul 07 15:44:45.775020 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:45.774963 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnl8" podStartSLOduration=2.064255632 podStartE2EDuration="5.774945579s" totalImagesPullingTime="3.710689947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:44:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:44:45.774219135 +0000 UTC m=+269.951863585" watchObservedRunningTime="2026-07-07 15:44:45.774945579 +0000 UTC m=+269.952590031" Jul 07 15:44:50.074438 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:50.074403 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm"] Jul 07 15:44:50.076731 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:50.076715 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm" Jul 07 15:44:50.078756 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:50.078730 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 15:44:50.078819 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:50.078776 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 15:44:50.078884 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:50.078861 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-s9d6n\"" Jul 07 15:44:50.088233 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:50.088212 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm"] Jul 07 15:44:50.207712 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:50.207674 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d23b589-682f-4ee2-9c0c-2905ec19b12f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm\" (UID: \"9d23b589-682f-4ee2-9c0c-2905ec19b12f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm" Jul 07 15:44:50.207873 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:50.207741 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d23b589-682f-4ee2-9c0c-2905ec19b12f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm\" (UID: \"9d23b589-682f-4ee2-9c0c-2905ec19b12f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm" Jul 07 15:44:50.207873 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:50.207806 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c9gz\" (UniqueName: \"kubernetes.io/projected/9d23b589-682f-4ee2-9c0c-2905ec19b12f-kube-api-access-8c9gz\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm\" (UID: \"9d23b589-682f-4ee2-9c0c-2905ec19b12f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm" Jul 07 15:44:50.308817 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:50.308783 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d23b589-682f-4ee2-9c0c-2905ec19b12f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm\" (UID: \"9d23b589-682f-4ee2-9c0c-2905ec19b12f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm" Jul 07 15:44:50.309003 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:50.308835 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9gz\" (UniqueName: \"kubernetes.io/projected/9d23b589-682f-4ee2-9c0c-2905ec19b12f-kube-api-access-8c9gz\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm\" (UID: \"9d23b589-682f-4ee2-9c0c-2905ec19b12f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm" Jul 07 15:44:50.309003 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:50.308860 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d23b589-682f-4ee2-9c0c-2905ec19b12f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm\" (UID: \"9d23b589-682f-4ee2-9c0c-2905ec19b12f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm" Jul 07 15:44:50.309320 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:50.309297 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d23b589-682f-4ee2-9c0c-2905ec19b12f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm\" (UID: \"9d23b589-682f-4ee2-9c0c-2905ec19b12f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm" Jul 07 15:44:50.309320 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:50.309315 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d23b589-682f-4ee2-9c0c-2905ec19b12f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm\" (UID: \"9d23b589-682f-4ee2-9c0c-2905ec19b12f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm" Jul 07 15:44:50.318380 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:50.318359 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c9gz\" (UniqueName: \"kubernetes.io/projected/9d23b589-682f-4ee2-9c0c-2905ec19b12f-kube-api-access-8c9gz\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm\" (UID: \"9d23b589-682f-4ee2-9c0c-2905ec19b12f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm" Jul 07 15:44:50.385912 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:50.385850 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm" Jul 07 15:44:50.520346 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:50.520311 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm"] Jul 07 15:44:50.523150 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:44:50.523106 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d23b589_682f_4ee2_9c0c_2905ec19b12f.slice/crio-2557d2a65d8eb74f8e6c47d545dbeb020cca04d2a2d74744d58a3e055b59187c WatchSource:0}: Error finding container 2557d2a65d8eb74f8e6c47d545dbeb020cca04d2a2d74744d58a3e055b59187c: Status 404 returned error can't find the container with id 2557d2a65d8eb74f8e6c47d545dbeb020cca04d2a2d74744d58a3e055b59187c Jul 07 15:44:50.784612 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:50.784581 2557 generic.go:356] "Generic (PLEG): container finished" podID="9d23b589-682f-4ee2-9c0c-2905ec19b12f" containerID="3e416b741ec238bb7e0a7775010bfc6a81e5c0e3caf2986440e0427eae6b69bd" exitCode=0 Jul 07 15:44:50.784914 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:50.784627 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm" event={"ID":"9d23b589-682f-4ee2-9c0c-2905ec19b12f","Type":"ContainerDied","Data":"3e416b741ec238bb7e0a7775010bfc6a81e5c0e3caf2986440e0427eae6b69bd"} Jul 07 15:44:50.784914 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:50.784661 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm" event={"ID":"9d23b589-682f-4ee2-9c0c-2905ec19b12f","Type":"ContainerStarted","Data":"2557d2a65d8eb74f8e6c47d545dbeb020cca04d2a2d74744d58a3e055b59187c"} Jul 07 15:44:51.790593 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:51.790558 2557 generic.go:356] "Generic (PLEG): container finished" podID="9d23b589-682f-4ee2-9c0c-2905ec19b12f" containerID="08e95e78bbe1799c34aed9ed34f5270b641f76adde13fc0f93a00455b0120bef" exitCode=0 Jul 07 15:44:51.791009 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:51.790637 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm" event={"ID":"9d23b589-682f-4ee2-9c0c-2905ec19b12f","Type":"ContainerDied","Data":"08e95e78bbe1799c34aed9ed34f5270b641f76adde13fc0f93a00455b0120bef"} Jul 07 15:44:52.797097 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:52.797063 2557 generic.go:356] "Generic (PLEG): container finished" podID="9d23b589-682f-4ee2-9c0c-2905ec19b12f" containerID="1d78a4184bb85e6a3d5baeac0c0e089897263361891594bffdafc3b56de42842" exitCode=0 Jul 07 15:44:52.797498 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:52.797149 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm" event={"ID":"9d23b589-682f-4ee2-9c0c-2905ec19b12f","Type":"ContainerDied","Data":"1d78a4184bb85e6a3d5baeac0c0e089897263361891594bffdafc3b56de42842"} Jul 07 15:44:53.933852 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:53.933830 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm" Jul 07 15:44:54.042320 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:54.042287 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d23b589-682f-4ee2-9c0c-2905ec19b12f-util\") pod \"9d23b589-682f-4ee2-9c0c-2905ec19b12f\" (UID: \"9d23b589-682f-4ee2-9c0c-2905ec19b12f\") " Jul 07 15:44:54.042527 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:54.042337 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d23b589-682f-4ee2-9c0c-2905ec19b12f-bundle\") pod \"9d23b589-682f-4ee2-9c0c-2905ec19b12f\" (UID: \"9d23b589-682f-4ee2-9c0c-2905ec19b12f\") " Jul 07 15:44:54.042527 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:54.042370 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c9gz\" (UniqueName: \"kubernetes.io/projected/9d23b589-682f-4ee2-9c0c-2905ec19b12f-kube-api-access-8c9gz\") pod \"9d23b589-682f-4ee2-9c0c-2905ec19b12f\" (UID: \"9d23b589-682f-4ee2-9c0c-2905ec19b12f\") " Jul 07 15:44:54.043215 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:54.043188 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d23b589-682f-4ee2-9c0c-2905ec19b12f-bundle" (OuterVolumeSpecName: "bundle") pod "9d23b589-682f-4ee2-9c0c-2905ec19b12f" (UID: "9d23b589-682f-4ee2-9c0c-2905ec19b12f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:44:54.044474 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:54.044440 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d23b589-682f-4ee2-9c0c-2905ec19b12f-kube-api-access-8c9gz" (OuterVolumeSpecName: "kube-api-access-8c9gz") pod "9d23b589-682f-4ee2-9c0c-2905ec19b12f" (UID: "9d23b589-682f-4ee2-9c0c-2905ec19b12f"). InnerVolumeSpecName "kube-api-access-8c9gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:44:54.047820 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:54.047785 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d23b589-682f-4ee2-9c0c-2905ec19b12f-util" (OuterVolumeSpecName: "util") pod "9d23b589-682f-4ee2-9c0c-2905ec19b12f" (UID: "9d23b589-682f-4ee2-9c0c-2905ec19b12f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:44:54.143973 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:54.143882 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d23b589-682f-4ee2-9c0c-2905ec19b12f-util\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:44:54.143973 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:54.143938 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d23b589-682f-4ee2-9c0c-2905ec19b12f-bundle\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:44:54.143973 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:54.143949 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c9gz\" (UniqueName: \"kubernetes.io/projected/9d23b589-682f-4ee2-9c0c-2905ec19b12f-kube-api-access-8c9gz\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:44:54.808947 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:54.808913 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm" Jul 07 15:44:54.808947 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:54.808933 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgbjzm" event={"ID":"9d23b589-682f-4ee2-9c0c-2905ec19b12f","Type":"ContainerDied","Data":"2557d2a65d8eb74f8e6c47d545dbeb020cca04d2a2d74744d58a3e055b59187c"} Jul 07 15:44:54.809254 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:44:54.808975 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2557d2a65d8eb74f8e6c47d545dbeb020cca04d2a2d74744d58a3e055b59187c" Jul 07 15:45:01.638708 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.638671 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8"] Jul 07 15:45:01.639164 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.638966 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d23b589-682f-4ee2-9c0c-2905ec19b12f" containerName="extract" Jul 07 15:45:01.639164 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.638976 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d23b589-682f-4ee2-9c0c-2905ec19b12f" containerName="extract" Jul 07 15:45:01.639164 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.638988 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d23b589-682f-4ee2-9c0c-2905ec19b12f" containerName="pull" Jul 07 15:45:01.639164 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.638994 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d23b589-682f-4ee2-9c0c-2905ec19b12f" containerName="pull" Jul 07 15:45:01.639164 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.639008 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d23b589-682f-4ee2-9c0c-2905ec19b12f" containerName="util" Jul 07 15:45:01.639164 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.639013 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d23b589-682f-4ee2-9c0c-2905ec19b12f" containerName="util" Jul 07 15:45:01.639164 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.639061 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d23b589-682f-4ee2-9c0c-2905ec19b12f" containerName="extract" Jul 07 15:45:01.641501 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.641484 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" Jul 07 15:45:01.649475 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.649452 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 15:45:01.649583 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.649526 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 15:45:01.649583 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.649541 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 15:45:01.649692 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.649579 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 15:45:01.649692 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.649606 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-z46jf\"" Jul 07 15:45:01.685271 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.685242 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8"] Jul 07 15:45:01.698459 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.698433 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3134dcbe-f349-45f2-9176-5cf7a2026c64-webhook-cert\") pod \"opendatahub-operator-controller-manager-77db849c7c-4wnc8\" (UID: \"3134dcbe-f349-45f2-9176-5cf7a2026c64\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" Jul 07 15:45:01.698562 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.698466 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3134dcbe-f349-45f2-9176-5cf7a2026c64-manifests\") pod \"opendatahub-operator-controller-manager-77db849c7c-4wnc8\" (UID: \"3134dcbe-f349-45f2-9176-5cf7a2026c64\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" Jul 07 15:45:01.698562 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.698512 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3134dcbe-f349-45f2-9176-5cf7a2026c64-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77db849c7c-4wnc8\" (UID: \"3134dcbe-f349-45f2-9176-5cf7a2026c64\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" Jul 07 15:45:01.698562 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.698549 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3134dcbe-f349-45f2-9176-5cf7a2026c64-tmp\") pod \"opendatahub-operator-controller-manager-77db849c7c-4wnc8\" (UID: \"3134dcbe-f349-45f2-9176-5cf7a2026c64\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" Jul 07 15:45:01.698664 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.698581 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5wgr\" (UniqueName: \"kubernetes.io/projected/3134dcbe-f349-45f2-9176-5cf7a2026c64-kube-api-access-z5wgr\") pod \"opendatahub-operator-controller-manager-77db849c7c-4wnc8\" (UID: \"3134dcbe-f349-45f2-9176-5cf7a2026c64\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" Jul 07 15:45:01.799401 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.799354 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3134dcbe-f349-45f2-9176-5cf7a2026c64-webhook-cert\") pod \"opendatahub-operator-controller-manager-77db849c7c-4wnc8\" (UID: \"3134dcbe-f349-45f2-9176-5cf7a2026c64\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" Jul 07 15:45:01.799401 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.799404 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3134dcbe-f349-45f2-9176-5cf7a2026c64-manifests\") pod \"opendatahub-operator-controller-manager-77db849c7c-4wnc8\" (UID: \"3134dcbe-f349-45f2-9176-5cf7a2026c64\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" Jul 07 15:45:01.799631 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.799599 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3134dcbe-f349-45f2-9176-5cf7a2026c64-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77db849c7c-4wnc8\" (UID: \"3134dcbe-f349-45f2-9176-5cf7a2026c64\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" Jul 07 15:45:01.799669 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.799653 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3134dcbe-f349-45f2-9176-5cf7a2026c64-tmp\") pod \"opendatahub-operator-controller-manager-77db849c7c-4wnc8\" (UID: \"3134dcbe-f349-45f2-9176-5cf7a2026c64\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" Jul 07 15:45:01.799705 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.799692 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5wgr\" (UniqueName: \"kubernetes.io/projected/3134dcbe-f349-45f2-9176-5cf7a2026c64-kube-api-access-z5wgr\") pod \"opendatahub-operator-controller-manager-77db849c7c-4wnc8\" (UID: \"3134dcbe-f349-45f2-9176-5cf7a2026c64\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" Jul 07 15:45:01.799777 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.799758 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3134dcbe-f349-45f2-9176-5cf7a2026c64-manifests\") pod \"opendatahub-operator-controller-manager-77db849c7c-4wnc8\" (UID: \"3134dcbe-f349-45f2-9176-5cf7a2026c64\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" Jul 07 15:45:01.801848 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.801818 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3134dcbe-f349-45f2-9176-5cf7a2026c64-tmp\") pod \"opendatahub-operator-controller-manager-77db849c7c-4wnc8\" (UID: \"3134dcbe-f349-45f2-9176-5cf7a2026c64\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" Jul 07 15:45:01.801957 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.801873 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3134dcbe-f349-45f2-9176-5cf7a2026c64-webhook-cert\") pod \"opendatahub-operator-controller-manager-77db849c7c-4wnc8\" (UID: \"3134dcbe-f349-45f2-9176-5cf7a2026c64\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" Jul 07 15:45:01.802034 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.802016 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3134dcbe-f349-45f2-9176-5cf7a2026c64-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77db849c7c-4wnc8\" (UID: \"3134dcbe-f349-45f2-9176-5cf7a2026c64\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" Jul 07 15:45:01.811150 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.811105 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5wgr\" (UniqueName: \"kubernetes.io/projected/3134dcbe-f349-45f2-9176-5cf7a2026c64-kube-api-access-z5wgr\") pod \"opendatahub-operator-controller-manager-77db849c7c-4wnc8\" (UID: \"3134dcbe-f349-45f2-9176-5cf7a2026c64\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" Jul 07 15:45:01.951069 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:01.950981 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" Jul 07 15:45:02.090287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:02.090146 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8"] Jul 07 15:45:02.093004 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:45:02.092963 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3134dcbe_f349_45f2_9176_5cf7a2026c64.slice/crio-f1795b7d6714b8838fa16752e93833d31cab1dbd31fa2e3c6422ae3aee693aa3 WatchSource:0}: Error finding container f1795b7d6714b8838fa16752e93833d31cab1dbd31fa2e3c6422ae3aee693aa3: Status 404 returned error can't find the container with id f1795b7d6714b8838fa16752e93833d31cab1dbd31fa2e3c6422ae3aee693aa3 Jul 07 15:45:02.845034 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:02.845000 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" event={"ID":"3134dcbe-f349-45f2-9176-5cf7a2026c64","Type":"ContainerStarted","Data":"f1795b7d6714b8838fa16752e93833d31cab1dbd31fa2e3c6422ae3aee693aa3"} Jul 07 15:45:05.860300 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:05.860258 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" event={"ID":"3134dcbe-f349-45f2-9176-5cf7a2026c64","Type":"ContainerStarted","Data":"e2fbb4b89d609ea5b600c163e5e3a8252d50be2d2ae63c2d1b3a8d967007e372"} Jul 07 15:45:06.866378 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:06.866289 2557 generic.go:356] "Generic (PLEG): container finished" podID="3134dcbe-f349-45f2-9176-5cf7a2026c64" containerID="e2fbb4b89d609ea5b600c163e5e3a8252d50be2d2ae63c2d1b3a8d967007e372" exitCode=0 Jul 07 15:45:06.866378 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:06.866348 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" event={"ID":"3134dcbe-f349-45f2-9176-5cf7a2026c64","Type":"ContainerDied","Data":"e2fbb4b89d609ea5b600c163e5e3a8252d50be2d2ae63c2d1b3a8d967007e372"} Jul 07 15:45:07.872878 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:07.872835 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" event={"ID":"3134dcbe-f349-45f2-9176-5cf7a2026c64","Type":"ContainerStarted","Data":"5f07f16369400a2ad812ee9a3bbc5b60236b1174bc40607be3227f0d56578136"} Jul 07 15:45:07.873371 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:07.872941 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" Jul 07 15:45:07.899684 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:07.899625 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" podStartSLOduration=1.886368249 podStartE2EDuration="6.899611084s" totalImagesPullingTime="3.922153408s" totalInitContainerRuntime="1.091089427s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:45:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:07.897699041 +0000 UTC m=+292.075343492" watchObservedRunningTime="2026-07-07 15:45:07.899611084 +0000 UTC m=+292.077255535" Jul 07 15:45:16.249363 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:16.249335 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/1.log" Jul 07 15:45:16.254663 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:16.254615 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hkj2z_07a80436-65b9-4f0d-99b3-ecda36240b48/ovn-acl-logging/0.log" Jul 07 15:45:16.256001 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:16.255975 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/1.log" Jul 07 15:45:16.261548 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:16.261523 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hkj2z_07a80436-65b9-4f0d-99b3-ecda36240b48/ovn-acl-logging/0.log" Jul 07 15:45:16.262690 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:16.262668 2557 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 15:45:18.880036 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:18.879969 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-4wnc8" Jul 07 15:45:22.800867 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:22.800826 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-d4cjs"] Jul 07 15:45:22.804783 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:22.804759 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-d4cjs" Jul 07 15:45:22.806950 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:22.806925 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 07 15:45:22.818584 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:22.818561 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-d4cjs"] Jul 07 15:45:22.881037 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:22.881003 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l72dc\" (UniqueName: \"kubernetes.io/projected/69ad1c34-3a8c-4fdd-b2f0-6182f7bf62b3-kube-api-access-l72dc\") pod \"model-serving-api-54d89bd5fc-d4cjs\" (UID: \"69ad1c34-3a8c-4fdd-b2f0-6182f7bf62b3\") " pod="opendatahub/model-serving-api-54d89bd5fc-d4cjs" Jul 07 15:45:22.881218 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:22.881052 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69ad1c34-3a8c-4fdd-b2f0-6182f7bf62b3-tls-certs\") pod \"model-serving-api-54d89bd5fc-d4cjs\" (UID: \"69ad1c34-3a8c-4fdd-b2f0-6182f7bf62b3\") " pod="opendatahub/model-serving-api-54d89bd5fc-d4cjs" Jul 07 15:45:22.982453 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:22.982413 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69ad1c34-3a8c-4fdd-b2f0-6182f7bf62b3-tls-certs\") pod \"model-serving-api-54d89bd5fc-d4cjs\" (UID: \"69ad1c34-3a8c-4fdd-b2f0-6182f7bf62b3\") " pod="opendatahub/model-serving-api-54d89bd5fc-d4cjs" Jul 07 15:45:22.982624 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:22.982491 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l72dc\" (UniqueName: \"kubernetes.io/projected/69ad1c34-3a8c-4fdd-b2f0-6182f7bf62b3-kube-api-access-l72dc\") pod \"model-serving-api-54d89bd5fc-d4cjs\" (UID: \"69ad1c34-3a8c-4fdd-b2f0-6182f7bf62b3\") " pod="opendatahub/model-serving-api-54d89bd5fc-d4cjs" Jul 07 15:45:22.984845 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:22.984818 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69ad1c34-3a8c-4fdd-b2f0-6182f7bf62b3-tls-certs\") pod \"model-serving-api-54d89bd5fc-d4cjs\" (UID: \"69ad1c34-3a8c-4fdd-b2f0-6182f7bf62b3\") " pod="opendatahub/model-serving-api-54d89bd5fc-d4cjs" Jul 07 15:45:22.991519 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:22.991485 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l72dc\" (UniqueName: \"kubernetes.io/projected/69ad1c34-3a8c-4fdd-b2f0-6182f7bf62b3-kube-api-access-l72dc\") pod \"model-serving-api-54d89bd5fc-d4cjs\" (UID: \"69ad1c34-3a8c-4fdd-b2f0-6182f7bf62b3\") " pod="opendatahub/model-serving-api-54d89bd5fc-d4cjs" Jul 07 15:45:23.115083 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:23.114995 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-d4cjs" Jul 07 15:45:23.257203 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:23.257093 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-d4cjs"] Jul 07 15:45:23.259430 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:45:23.259395 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ad1c34_3a8c_4fdd_b2f0_6182f7bf62b3.slice/crio-ca5a2614be13551acf580d052f3cbd7e6d61c4fed9bd102eae733dd290691d68 WatchSource:0}: Error finding container ca5a2614be13551acf580d052f3cbd7e6d61c4fed9bd102eae733dd290691d68: Status 404 returned error can't find the container with id ca5a2614be13551acf580d052f3cbd7e6d61c4fed9bd102eae733dd290691d68 Jul 07 15:45:23.261048 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:23.261032 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 15:45:23.953245 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:23.953212 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-d4cjs" event={"ID":"69ad1c34-3a8c-4fdd-b2f0-6182f7bf62b3","Type":"ContainerStarted","Data":"ca5a2614be13551acf580d052f3cbd7e6d61c4fed9bd102eae733dd290691d68"} Jul 07 15:45:25.965309 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:25.965271 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-d4cjs" event={"ID":"69ad1c34-3a8c-4fdd-b2f0-6182f7bf62b3","Type":"ContainerStarted","Data":"60d2c3c42e8424fe304900b92d2216fa8a2458f9cf7b6cb17260485b3af40919"} Jul 07 15:45:25.965722 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:25.965408 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-d4cjs" Jul 07 15:45:25.989398 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:25.989336 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-d4cjs" podStartSLOduration=2.008692271 podStartE2EDuration="3.989316383s" totalImagesPullingTime="1.980624112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:25.988863709 +0000 UTC m=+310.166508159" watchObservedRunningTime="2026-07-07 15:45:25.989316383 +0000 UTC m=+310.166960835" Jul 07 15:45:27.336527 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:27.336494 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6669df4868-vm6h8"] Jul 07 15:45:27.338926 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:27.338907 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6669df4868-vm6h8" Jul 07 15:45:27.341064 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:27.341043 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 07 15:45:27.341343 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:27.341327 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 15:45:27.341568 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:27.341550 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5c8jv\"" Jul 07 15:45:27.354736 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:27.354714 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6669df4868-vm6h8"] Jul 07 15:45:27.418274 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:27.418244 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e096f9-6fb3-4444-ae6b-60daf719f0de-tls-certs\") pod \"kube-auth-proxy-6669df4868-vm6h8\" (UID: \"00e096f9-6fb3-4444-ae6b-60daf719f0de\") " pod="openshift-ingress/kube-auth-proxy-6669df4868-vm6h8" Jul 07 15:45:27.418274 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:27.418279 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00e096f9-6fb3-4444-ae6b-60daf719f0de-tmp\") pod \"kube-auth-proxy-6669df4868-vm6h8\" (UID: \"00e096f9-6fb3-4444-ae6b-60daf719f0de\") " pod="openshift-ingress/kube-auth-proxy-6669df4868-vm6h8" Jul 07 15:45:27.418521 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:27.418403 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfzpd\" (UniqueName: \"kubernetes.io/projected/00e096f9-6fb3-4444-ae6b-60daf719f0de-kube-api-access-cfzpd\") pod \"kube-auth-proxy-6669df4868-vm6h8\" (UID: \"00e096f9-6fb3-4444-ae6b-60daf719f0de\") " pod="openshift-ingress/kube-auth-proxy-6669df4868-vm6h8" Jul 07 15:45:27.519799 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:27.519758 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfzpd\" (UniqueName: \"kubernetes.io/projected/00e096f9-6fb3-4444-ae6b-60daf719f0de-kube-api-access-cfzpd\") pod \"kube-auth-proxy-6669df4868-vm6h8\" (UID: \"00e096f9-6fb3-4444-ae6b-60daf719f0de\") " pod="openshift-ingress/kube-auth-proxy-6669df4868-vm6h8" Jul 07 15:45:27.520010 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:27.519820 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e096f9-6fb3-4444-ae6b-60daf719f0de-tls-certs\") pod \"kube-auth-proxy-6669df4868-vm6h8\" (UID: \"00e096f9-6fb3-4444-ae6b-60daf719f0de\") " pod="openshift-ingress/kube-auth-proxy-6669df4868-vm6h8" Jul 07 15:45:27.520010 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:45:27.519909 2557 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 07 15:45:27.520010 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:45:27.519957 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00e096f9-6fb3-4444-ae6b-60daf719f0de-tls-certs podName:00e096f9-6fb3-4444-ae6b-60daf719f0de nodeName:}" failed. No retries permitted until 2026-07-07 15:45:28.01994093 +0000 UTC m=+312.197585358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/00e096f9-6fb3-4444-ae6b-60daf719f0de-tls-certs") pod "kube-auth-proxy-6669df4868-vm6h8" (UID: "00e096f9-6fb3-4444-ae6b-60daf719f0de") : secret "kube-auth-proxy-tls" not found Jul 07 15:45:27.520010 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:27.519994 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00e096f9-6fb3-4444-ae6b-60daf719f0de-tmp\") pod \"kube-auth-proxy-6669df4868-vm6h8\" (UID: \"00e096f9-6fb3-4444-ae6b-60daf719f0de\") " pod="openshift-ingress/kube-auth-proxy-6669df4868-vm6h8" Jul 07 15:45:27.522341 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:27.522312 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00e096f9-6fb3-4444-ae6b-60daf719f0de-tmp\") pod \"kube-auth-proxy-6669df4868-vm6h8\" (UID: \"00e096f9-6fb3-4444-ae6b-60daf719f0de\") " pod="openshift-ingress/kube-auth-proxy-6669df4868-vm6h8" Jul 07 15:45:27.530371 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:27.530346 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfzpd\" (UniqueName: \"kubernetes.io/projected/00e096f9-6fb3-4444-ae6b-60daf719f0de-kube-api-access-cfzpd\") pod \"kube-auth-proxy-6669df4868-vm6h8\" (UID: \"00e096f9-6fb3-4444-ae6b-60daf719f0de\") " pod="openshift-ingress/kube-auth-proxy-6669df4868-vm6h8" Jul 07 15:45:28.024847 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:28.024816 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e096f9-6fb3-4444-ae6b-60daf719f0de-tls-certs\") pod \"kube-auth-proxy-6669df4868-vm6h8\" (UID: \"00e096f9-6fb3-4444-ae6b-60daf719f0de\") " pod="openshift-ingress/kube-auth-proxy-6669df4868-vm6h8" Jul 07 15:45:28.027271 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:28.027244 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e096f9-6fb3-4444-ae6b-60daf719f0de-tls-certs\") pod \"kube-auth-proxy-6669df4868-vm6h8\" (UID: \"00e096f9-6fb3-4444-ae6b-60daf719f0de\") " pod="openshift-ingress/kube-auth-proxy-6669df4868-vm6h8" Jul 07 15:45:28.248933 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:28.248906 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6669df4868-vm6h8" Jul 07 15:45:28.414031 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:28.413937 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6669df4868-vm6h8"] Jul 07 15:45:28.421559 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:45:28.421532 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e096f9_6fb3_4444_ae6b_60daf719f0de.slice/crio-9ccd5899836493bd2eb866c38afa2a2f2aed11b008b795c5d367af94fb9ae218 WatchSource:0}: Error finding container 9ccd5899836493bd2eb866c38afa2a2f2aed11b008b795c5d367af94fb9ae218: Status 404 returned error can't find the container with id 9ccd5899836493bd2eb866c38afa2a2f2aed11b008b795c5d367af94fb9ae218 Jul 07 15:45:28.980910 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:28.980873 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6669df4868-vm6h8" event={"ID":"00e096f9-6fb3-4444-ae6b-60daf719f0de","Type":"ContainerStarted","Data":"9ccd5899836493bd2eb866c38afa2a2f2aed11b008b795c5d367af94fb9ae218"} Jul 07 15:45:29.050644 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.050607 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-d47bp"] Jul 07 15:45:29.053263 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.053240 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.055919 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.055898 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 07 15:45:29.056068 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.056049 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 07 15:45:29.056165 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.056101 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 07 15:45:29.056325 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.056305 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7nvd4\"" Jul 07 15:45:29.056522 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.056509 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 15:45:29.076504 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.076466 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-d47bp"] Jul 07 15:45:29.134946 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.134911 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-token\") pod \"istiod-openshift-gateway-7574459656-d47bp\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.135168 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.134963 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-d47bp\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.135168 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.135045 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-local-certs\") pod \"istiod-openshift-gateway-7574459656-d47bp\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.135168 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.135076 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcc7w\" (UniqueName: \"kubernetes.io/projected/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-kube-api-access-gcc7w\") pod \"istiod-openshift-gateway-7574459656-d47bp\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.135168 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.135133 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-d47bp\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.135397 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.135176 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-d47bp\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.135397 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.135211 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-cacerts\") pod \"istiod-openshift-gateway-7574459656-d47bp\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.236235 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.236152 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-local-certs\") pod \"istiod-openshift-gateway-7574459656-d47bp\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.236235 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.236194 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcc7w\" (UniqueName: \"kubernetes.io/projected/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-kube-api-access-gcc7w\") pod \"istiod-openshift-gateway-7574459656-d47bp\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.236235 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.236220 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-d47bp\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.236509 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.236257 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-d47bp\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.236509 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.236299 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-cacerts\") pod \"istiod-openshift-gateway-7574459656-d47bp\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.236509 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.236331 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-token\") pod \"istiod-openshift-gateway-7574459656-d47bp\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.236509 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.236378 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-d47bp\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.236927 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.236899 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-d47bp\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.238593 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.238550 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-local-certs\") pod \"istiod-openshift-gateway-7574459656-d47bp\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.238825 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.238800 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-cacerts\") pod \"istiod-openshift-gateway-7574459656-d47bp\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.238903 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.238814 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-d47bp\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.238903 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.238864 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-d47bp\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.249191 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.249170 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-token\") pod \"istiod-openshift-gateway-7574459656-d47bp\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.254545 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.254485 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcc7w\" (UniqueName: \"kubernetes.io/projected/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-kube-api-access-gcc7w\") pod \"istiod-openshift-gateway-7574459656-d47bp\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.362632 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.362593 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:29.537562 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.537532 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-d47bp"] Jul 07 15:45:29.539817 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:45:29.539785 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b924fcb_cec0_4e00_afbd_f30a1b9b5c09.slice/crio-6adf859259367e60e8064a56976905df83298a03a2ae17309ceb97d8c5f0ba21 WatchSource:0}: Error finding container 6adf859259367e60e8064a56976905df83298a03a2ae17309ceb97d8c5f0ba21: Status 404 returned error can't find the container with id 6adf859259367e60e8064a56976905df83298a03a2ae17309ceb97d8c5f0ba21 Jul 07 15:45:29.989810 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:29.989725 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" event={"ID":"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09","Type":"ContainerStarted","Data":"6adf859259367e60e8064a56976905df83298a03a2ae17309ceb97d8c5f0ba21"} Jul 07 15:45:32.329191 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:32.329153 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 15:45:32.329439 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:32.329233 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 15:45:33.008321 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.008279 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" event={"ID":"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09","Type":"ContainerStarted","Data":"b3571abe1cf2a9edf468a5b060bcd162f29e82222fa8b639f9a615d1095a78b2"} Jul 07 15:45:33.008525 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.008345 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:33.010923 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.010870 2557 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-d47bp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 15:45:33.011070 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.010939 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" podUID="0b924fcb-cec0-4e00-afbd-f30a1b9b5c09" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 15:45:33.011164 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.011131 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6669df4868-vm6h8" event={"ID":"00e096f9-6fb3-4444-ae6b-60daf719f0de","Type":"ContainerStarted","Data":"6da2111d0b351811f529d9b024b46d6ca4aaeb24a4024c40ade5d7829f9a11eb"} Jul 07 15:45:33.062068 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.062012 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" podStartSLOduration=1.275521715 podStartE2EDuration="4.061995292s" totalImagesPullingTime="2.786473577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:45:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:33.056722817 +0000 UTC m=+317.234367285" watchObservedRunningTime="2026-07-07 15:45:33.061995292 +0000 UTC m=+317.239639742" Jul 07 15:45:33.133077 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.133013 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6669df4868-vm6h8" podStartSLOduration=2.229126362 podStartE2EDuration="6.132968024s" totalImagesPullingTime="3.903841662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:33.129228288 +0000 UTC m=+317.306872760" watchObservedRunningTime="2026-07-07 15:45:33.132968024 +0000 UTC m=+317.310612474" Jul 07 15:45:33.467877 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.467839 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6h895"] Jul 07 15:45:33.471669 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.471645 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6h895" Jul 07 15:45:33.473674 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.473654 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-xxd5p\"" Jul 07 15:45:33.473795 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.473743 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 07 15:45:33.487519 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.487496 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxdl"] Jul 07 15:45:33.490687 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.490667 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6h895"] Jul 07 15:45:33.490789 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.490765 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxdl" Jul 07 15:45:33.493941 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.493922 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 07 15:45:33.494473 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.494452 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-pk4mq\"" Jul 07 15:45:33.506627 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.506605 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxdl"] Jul 07 15:45:33.521186 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.521153 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t"] Jul 07 15:45:33.530806 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.529604 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.532591 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.532413 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-mlhb6\"" Jul 07 15:45:33.536020 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.535909 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t"] Jul 07 15:45:33.585134 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.583685 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmzqs\" (UniqueName: \"kubernetes.io/projected/34784301-08df-4c57-b141-07a592143554-kube-api-access-gmzqs\") pod \"kserve-controller-manager-d6f4b65b8-6h895\" (UID: \"34784301-08df-4c57-b141-07a592143554\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6h895" Jul 07 15:45:33.585134 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.583756 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24czc\" (UniqueName: \"kubernetes.io/projected/3ed630e1-4529-4df0-840a-748a59a09b1a-kube-api-access-24czc\") pod \"llmisvc-controller-manager-777c6bd7d5-9qxdl\" (UID: \"3ed630e1-4529-4df0-840a-748a59a09b1a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxdl" Jul 07 15:45:33.585134 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.583930 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34784301-08df-4c57-b141-07a592143554-cert\") pod \"kserve-controller-manager-d6f4b65b8-6h895\" (UID: \"34784301-08df-4c57-b141-07a592143554\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6h895" Jul 07 15:45:33.585134 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.583981 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ed630e1-4529-4df0-840a-748a59a09b1a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9qxdl\" (UID: \"3ed630e1-4529-4df0-840a-748a59a09b1a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxdl" Jul 07 15:45:33.685199 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.685161 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7607df7a-3a72-46ef-b035-74355b31932a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.685396 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.685212 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf9jc\" (UniqueName: \"kubernetes.io/projected/7607df7a-3a72-46ef-b035-74355b31932a-kube-api-access-zf9jc\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.685396 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.685237 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34784301-08df-4c57-b141-07a592143554-cert\") pod \"kserve-controller-manager-d6f4b65b8-6h895\" (UID: \"34784301-08df-4c57-b141-07a592143554\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6h895" Jul 07 15:45:33.685396 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.685265 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7607df7a-3a72-46ef-b035-74355b31932a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.685396 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.685300 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7607df7a-3a72-46ef-b035-74355b31932a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.685633 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.685420 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7607df7a-3a72-46ef-b035-74355b31932a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.685633 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.685453 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ed630e1-4529-4df0-840a-748a59a09b1a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9qxdl\" (UID: \"3ed630e1-4529-4df0-840a-748a59a09b1a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxdl" Jul 07 15:45:33.685633 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.685514 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmzqs\" (UniqueName: \"kubernetes.io/projected/34784301-08df-4c57-b141-07a592143554-kube-api-access-gmzqs\") pod \"kserve-controller-manager-d6f4b65b8-6h895\" (UID: \"34784301-08df-4c57-b141-07a592143554\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6h895" Jul 07 15:45:33.685633 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.685544 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24czc\" (UniqueName: \"kubernetes.io/projected/3ed630e1-4529-4df0-840a-748a59a09b1a-kube-api-access-24czc\") pod \"llmisvc-controller-manager-777c6bd7d5-9qxdl\" (UID: \"3ed630e1-4529-4df0-840a-748a59a09b1a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxdl" Jul 07 15:45:33.685633 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:45:33.685587 2557 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 07 15:45:33.685633 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.685621 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7607df7a-3a72-46ef-b035-74355b31932a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.685941 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:45:33.685672 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed630e1-4529-4df0-840a-748a59a09b1a-cert podName:3ed630e1-4529-4df0-840a-748a59a09b1a nodeName:}" failed. No retries permitted until 2026-07-07 15:45:34.185650196 +0000 UTC m=+318.363294630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3ed630e1-4529-4df0-840a-748a59a09b1a-cert") pod "llmisvc-controller-manager-777c6bd7d5-9qxdl" (UID: "3ed630e1-4529-4df0-840a-748a59a09b1a") : secret "llmisvc-webhook-server-cert" not found Jul 07 15:45:33.685941 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.685765 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7607df7a-3a72-46ef-b035-74355b31932a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.685941 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.685831 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7607df7a-3a72-46ef-b035-74355b31932a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.685941 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.685856 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7607df7a-3a72-46ef-b035-74355b31932a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.687901 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.687878 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34784301-08df-4c57-b141-07a592143554-cert\") pod \"kserve-controller-manager-d6f4b65b8-6h895\" (UID: \"34784301-08df-4c57-b141-07a592143554\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6h895" Jul 07 15:45:33.697671 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.697644 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24czc\" (UniqueName: \"kubernetes.io/projected/3ed630e1-4529-4df0-840a-748a59a09b1a-kube-api-access-24czc\") pod \"llmisvc-controller-manager-777c6bd7d5-9qxdl\" (UID: \"3ed630e1-4529-4df0-840a-748a59a09b1a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxdl" Jul 07 15:45:33.726208 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.726153 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmzqs\" (UniqueName: \"kubernetes.io/projected/34784301-08df-4c57-b141-07a592143554-kube-api-access-gmzqs\") pod \"kserve-controller-manager-d6f4b65b8-6h895\" (UID: \"34784301-08df-4c57-b141-07a592143554\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6h895" Jul 07 15:45:33.782270 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.782241 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6h895" Jul 07 15:45:33.786725 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.786691 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7607df7a-3a72-46ef-b035-74355b31932a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.786834 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.786739 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7607df7a-3a72-46ef-b035-74355b31932a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.786834 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.786785 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7607df7a-3a72-46ef-b035-74355b31932a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.786834 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.786812 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7607df7a-3a72-46ef-b035-74355b31932a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.787106 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.787084 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7607df7a-3a72-46ef-b035-74355b31932a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.787207 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.787147 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf9jc\" (UniqueName: \"kubernetes.io/projected/7607df7a-3a72-46ef-b035-74355b31932a-kube-api-access-zf9jc\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.787207 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.787196 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7607df7a-3a72-46ef-b035-74355b31932a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.787321 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.787224 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7607df7a-3a72-46ef-b035-74355b31932a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.787321 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.787244 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7607df7a-3a72-46ef-b035-74355b31932a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.787321 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.787259 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7607df7a-3a72-46ef-b035-74355b31932a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.787469 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.787331 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7607df7a-3a72-46ef-b035-74355b31932a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.787646 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.787623 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7607df7a-3a72-46ef-b035-74355b31932a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.787891 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.787870 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7607df7a-3a72-46ef-b035-74355b31932a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.788042 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.788020 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7607df7a-3a72-46ef-b035-74355b31932a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.789494 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.789476 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7607df7a-3a72-46ef-b035-74355b31932a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.789797 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.789781 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7607df7a-3a72-46ef-b035-74355b31932a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.804063 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.804004 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf9jc\" (UniqueName: \"kubernetes.io/projected/7607df7a-3a72-46ef-b035-74355b31932a-kube-api-access-zf9jc\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.804216 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.804198 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7607df7a-3a72-46ef-b035-74355b31932a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4smx5t\" (UID: \"7607df7a-3a72-46ef-b035-74355b31932a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.846620 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.846323 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:33.939509 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:33.939412 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6h895"] Jul 07 15:45:33.941381 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:45:33.941350 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34784301_08df_4c57_b141_07a592143554.slice/crio-f354d4d4e47a9f41f999e6c8ec0806d5d7d72ceab7ba675c5830bb59922f34b8 WatchSource:0}: Error finding container f354d4d4e47a9f41f999e6c8ec0806d5d7d72ceab7ba675c5830bb59922f34b8: Status 404 returned error can't find the container with id f354d4d4e47a9f41f999e6c8ec0806d5d7d72ceab7ba675c5830bb59922f34b8 Jul 07 15:45:34.018413 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:34.018319 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6h895" event={"ID":"34784301-08df-4c57-b141-07a592143554","Type":"ContainerStarted","Data":"f354d4d4e47a9f41f999e6c8ec0806d5d7d72ceab7ba675c5830bb59922f34b8"} Jul 07 15:45:34.019327 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:34.019303 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:34.020606 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:34.020585 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t"] Jul 07 15:45:34.022054 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:45:34.022018 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7607df7a_3a72_46ef_b035_74355b31932a.slice/crio-63db6f69e850f1bd13827f8880c964c1b0354c7cf834e611db2a7d776dc9fa46 WatchSource:0}: Error finding container 63db6f69e850f1bd13827f8880c964c1b0354c7cf834e611db2a7d776dc9fa46: Status 404 returned error can't find the container with id 63db6f69e850f1bd13827f8880c964c1b0354c7cf834e611db2a7d776dc9fa46 Jul 07 15:45:34.192213 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:34.192185 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ed630e1-4529-4df0-840a-748a59a09b1a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9qxdl\" (UID: \"3ed630e1-4529-4df0-840a-748a59a09b1a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxdl" Jul 07 15:45:34.194538 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:34.194513 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ed630e1-4529-4df0-840a-748a59a09b1a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9qxdl\" (UID: \"3ed630e1-4529-4df0-840a-748a59a09b1a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxdl" Jul 07 15:45:34.403663 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:34.403635 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxdl" Jul 07 15:45:34.538325 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:34.538299 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxdl"] Jul 07 15:45:34.539976 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:45:34.539951 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3ed630e1_4529_4df0_840a_748a59a09b1a.slice/crio-3cbfc867873fbe7fef520da6db1b4627e8ad4844828380ec173fa2f1351879ee WatchSource:0}: Error finding container 3cbfc867873fbe7fef520da6db1b4627e8ad4844828380ec173fa2f1351879ee: Status 404 returned error can't find the container with id 3cbfc867873fbe7fef520da6db1b4627e8ad4844828380ec173fa2f1351879ee Jul 07 15:45:35.027151 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:35.027079 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxdl" event={"ID":"3ed630e1-4529-4df0-840a-748a59a09b1a","Type":"ContainerStarted","Data":"3cbfc867873fbe7fef520da6db1b4627e8ad4844828380ec173fa2f1351879ee"} Jul 07 15:45:35.028722 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:35.028693 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" event={"ID":"7607df7a-3a72-46ef-b035-74355b31932a","Type":"ContainerStarted","Data":"63db6f69e850f1bd13827f8880c964c1b0354c7cf834e611db2a7d776dc9fa46"} Jul 07 15:45:35.934733 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:35.934704 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-d47bp"] Jul 07 15:45:36.035410 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:36.035263 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" podUID="0b924fcb-cec0-4e00-afbd-f30a1b9b5c09" containerName="discovery" containerID="cri-o://b3571abe1cf2a9edf468a5b060bcd162f29e82222fa8b639f9a615d1095a78b2" gracePeriod=30 Jul 07 15:45:36.974311 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:36.974230 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-d4cjs" Jul 07 15:45:37.044360 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:37.044323 2557 generic.go:356] "Generic (PLEG): container finished" podID="0b924fcb-cec0-4e00-afbd-f30a1b9b5c09" containerID="b3571abe1cf2a9edf468a5b060bcd162f29e82222fa8b639f9a615d1095a78b2" exitCode=0 Jul 07 15:45:37.044544 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:37.044404 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" event={"ID":"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09","Type":"ContainerDied","Data":"b3571abe1cf2a9edf468a5b060bcd162f29e82222fa8b639f9a615d1095a78b2"} Jul 07 15:45:37.854954 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:37.854931 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:38.029844 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.029801 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-csr-dns-cert\") pod \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " Jul 07 15:45:38.029844 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.029848 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-cacerts\") pod \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " Jul 07 15:45:38.030397 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.029915 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcc7w\" (UniqueName: \"kubernetes.io/projected/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-kube-api-access-gcc7w\") pod \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " Jul 07 15:45:38.030397 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.029947 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-csr-ca-configmap\") pod \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " Jul 07 15:45:38.030397 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.029978 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-kubeconfig\") pod \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " Jul 07 15:45:38.030397 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.030069 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-local-certs\") pod \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " Jul 07 15:45:38.030397 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.030096 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-token\") pod \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\" (UID: \"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09\") " Jul 07 15:45:38.030741 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.030672 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "0b924fcb-cec0-4e00-afbd-f30a1b9b5c09" (UID: "0b924fcb-cec0-4e00-afbd-f30a1b9b5c09"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:45:38.032763 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.032736 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "0b924fcb-cec0-4e00-afbd-f30a1b9b5c09" (UID: "0b924fcb-cec0-4e00-afbd-f30a1b9b5c09"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:45:38.032935 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.032914 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-cacerts" (OuterVolumeSpecName: "cacerts") pod "0b924fcb-cec0-4e00-afbd-f30a1b9b5c09" (UID: "0b924fcb-cec0-4e00-afbd-f30a1b9b5c09"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:45:38.033044 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.033014 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "0b924fcb-cec0-4e00-afbd-f30a1b9b5c09" (UID: "0b924fcb-cec0-4e00-afbd-f30a1b9b5c09"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:45:38.033351 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.033325 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b924fcb-cec0-4e00-afbd-f30a1b9b5c09" (UID: "0b924fcb-cec0-4e00-afbd-f30a1b9b5c09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:45:38.033630 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.033603 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-local-certs" (OuterVolumeSpecName: "local-certs") pod "0b924fcb-cec0-4e00-afbd-f30a1b9b5c09" (UID: "0b924fcb-cec0-4e00-afbd-f30a1b9b5c09"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:45:38.033703 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.033606 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-kube-api-access-gcc7w" (OuterVolumeSpecName: "kube-api-access-gcc7w") pod "0b924fcb-cec0-4e00-afbd-f30a1b9b5c09" (UID: "0b924fcb-cec0-4e00-afbd-f30a1b9b5c09"). InnerVolumeSpecName "kube-api-access-gcc7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:45:38.052548 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.052514 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" event={"ID":"0b924fcb-cec0-4e00-afbd-f30a1b9b5c09","Type":"ContainerDied","Data":"6adf859259367e60e8064a56976905df83298a03a2ae17309ceb97d8c5f0ba21"} Jul 07 15:45:38.052650 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.052524 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" Jul 07 15:45:38.052729 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.052629 2557 scope.go:117] "RemoveContainer" containerID="b3571abe1cf2a9edf468a5b060bcd162f29e82222fa8b639f9a615d1095a78b2" Jul 07 15:45:38.086504 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.086443 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-d47bp"] Jul 07 15:45:38.100591 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.100562 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-d47bp"] Jul 07 15:45:38.131735 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.131710 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-token\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:45:38.131735 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.131732 2557 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-csr-dns-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:45:38.131884 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.131746 2557 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-cacerts\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:45:38.131884 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.131760 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcc7w\" (UniqueName: \"kubernetes.io/projected/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-kube-api-access-gcc7w\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:45:38.131884 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.131769 2557 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-csr-ca-configmap\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:45:38.131884 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.131778 2557 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-istio-kubeconfig\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:45:38.131884 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.131786 2557 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09-local-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:45:38.353347 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.353265 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b924fcb-cec0-4e00-afbd-f30a1b9b5c09" path="/var/lib/kubelet/pods/0b924fcb-cec0-4e00-afbd-f30a1b9b5c09/volumes" Jul 07 15:45:38.470545 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.470491 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 15:45:38.470697 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.470567 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 15:45:38.470697 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:38.470604 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 07 15:45:39.059342 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:39.059311 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxdl" event={"ID":"3ed630e1-4529-4df0-840a-748a59a09b1a","Type":"ContainerStarted","Data":"976a2c33e7d0357edb7b0673a986907228d9761f2bd0c6f3573e40c6e1d34471"} Jul 07 15:45:39.059785 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:39.059459 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxdl" Jul 07 15:45:39.060884 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:39.060862 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6h895" event={"ID":"34784301-08df-4c57-b141-07a592143554","Type":"ContainerStarted","Data":"86abb825c9bbb56dd2ed96f71134c0567626151ac359525aec59ab677cd0dee7"} Jul 07 15:45:39.060958 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:39.060908 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6h895" Jul 07 15:45:39.063323 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:39.063301 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" event={"ID":"7607df7a-3a72-46ef-b035-74355b31932a","Type":"ContainerStarted","Data":"14165ca0f62a16902adfcc7e2c14694e3df01436e17f5c0e8c96497cee01ff39"} Jul 07 15:45:39.090815 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:39.090774 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxdl" podStartSLOduration=2.106204687 podStartE2EDuration="6.090760219s" totalImagesPullingTime="3.984555532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:39.089028439 +0000 UTC m=+323.266672889" watchObservedRunningTime="2026-07-07 15:45:39.090760219 +0000 UTC m=+323.268404686" Jul 07 15:45:39.119858 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:39.119810 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" podStartSLOduration=1.6737444849999998 podStartE2EDuration="6.119795595s" totalImagesPullingTime="4.44605111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:39.117611258 +0000 UTC m=+323.295255708" watchObservedRunningTime="2026-07-07 15:45:39.119795595 +0000 UTC m=+323.297440045" Jul 07 15:45:39.151801 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:39.151758 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6h895" podStartSLOduration=1.625762194 podStartE2EDuration="6.151745498s" totalImagesPullingTime="4.525983304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:39.147106743 +0000 UTC m=+323.324751193" watchObservedRunningTime="2026-07-07 15:45:39.151745498 +0000 UTC m=+323.329389930" Jul 07 15:45:39.847319 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:39.847280 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:39.851869 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:39.851847 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:40.069589 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:40.069545 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:40.070191 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:40.070174 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4smx5t" Jul 07 15:45:42.019934 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:42.019895 2557 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-d47bp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 07 15:45:42.020335 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:45:42.019960 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d47bp" podUID="0b924fcb-cec0-4e00-afbd-f30a1b9b5c09" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.35:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 15:46:10.071315 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:10.071220 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxdl" Jul 07 15:46:10.074260 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:10.074203 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6h895" Jul 07 15:46:23.013458 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:23.013419 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rgj79"] Jul 07 15:46:23.013863 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:23.013755 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b924fcb-cec0-4e00-afbd-f30a1b9b5c09" containerName="discovery" Jul 07 15:46:23.013863 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:23.013766 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b924fcb-cec0-4e00-afbd-f30a1b9b5c09" containerName="discovery" Jul 07 15:46:23.013863 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:23.013849 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b924fcb-cec0-4e00-afbd-f30a1b9b5c09" containerName="discovery" Jul 07 15:46:23.016762 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:23.016744 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rgj79" Jul 07 15:46:23.019350 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:23.019322 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 15:46:23.019487 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:23.019322 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-7kh4j\"" Jul 07 15:46:23.019487 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:23.019325 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 15:46:23.029760 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:23.029717 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rgj79"] Jul 07 15:46:23.118512 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:23.118475 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwx8f\" (UniqueName: \"kubernetes.io/projected/c4a79fe8-363d-4b53-8e81-758d9b404545-kube-api-access-fwx8f\") pod \"kuadrant-operator-catalog-rgj79\" (UID: \"c4a79fe8-363d-4b53-8e81-758d9b404545\") " pod="kuadrant-system/kuadrant-operator-catalog-rgj79" Jul 07 15:46:23.219217 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:23.219182 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwx8f\" (UniqueName: \"kubernetes.io/projected/c4a79fe8-363d-4b53-8e81-758d9b404545-kube-api-access-fwx8f\") pod \"kuadrant-operator-catalog-rgj79\" (UID: \"c4a79fe8-363d-4b53-8e81-758d9b404545\") " pod="kuadrant-system/kuadrant-operator-catalog-rgj79" Jul 07 15:46:23.227377 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:23.227349 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwx8f\" (UniqueName: \"kubernetes.io/projected/c4a79fe8-363d-4b53-8e81-758d9b404545-kube-api-access-fwx8f\") pod \"kuadrant-operator-catalog-rgj79\" (UID: \"c4a79fe8-363d-4b53-8e81-758d9b404545\") " pod="kuadrant-system/kuadrant-operator-catalog-rgj79" Jul 07 15:46:23.326807 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:23.326776 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rgj79" Jul 07 15:46:23.378702 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:23.378677 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rgj79"] Jul 07 15:46:23.462189 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:23.462164 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rgj79"] Jul 07 15:46:23.464790 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:46:23.464765 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a79fe8_363d_4b53_8e81_758d9b404545.slice/crio-7617f2cf96a7812f50a6045666ad3b9e7f3347e0b167edf508b9793235f21da1 WatchSource:0}: Error finding container 7617f2cf96a7812f50a6045666ad3b9e7f3347e0b167edf508b9793235f21da1: Status 404 returned error can't find the container with id 7617f2cf96a7812f50a6045666ad3b9e7f3347e0b167edf508b9793235f21da1 Jul 07 15:46:23.588805 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:23.588724 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6lgzr"] Jul 07 15:46:23.591385 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:23.591368 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6lgzr" Jul 07 15:46:23.601130 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:23.601095 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6lgzr"] Jul 07 15:46:23.722904 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:23.722874 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66vmk\" (UniqueName: \"kubernetes.io/projected/ec46ef29-6d38-44fd-a742-e748bec0ffe6-kube-api-access-66vmk\") pod \"kuadrant-operator-catalog-6lgzr\" (UID: \"ec46ef29-6d38-44fd-a742-e748bec0ffe6\") " pod="kuadrant-system/kuadrant-operator-catalog-6lgzr" Jul 07 15:46:23.824016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:23.823984 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66vmk\" (UniqueName: \"kubernetes.io/projected/ec46ef29-6d38-44fd-a742-e748bec0ffe6-kube-api-access-66vmk\") pod \"kuadrant-operator-catalog-6lgzr\" (UID: \"ec46ef29-6d38-44fd-a742-e748bec0ffe6\") " pod="kuadrant-system/kuadrant-operator-catalog-6lgzr" Jul 07 15:46:23.835592 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:23.835563 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66vmk\" (UniqueName: \"kubernetes.io/projected/ec46ef29-6d38-44fd-a742-e748bec0ffe6-kube-api-access-66vmk\") pod \"kuadrant-operator-catalog-6lgzr\" (UID: \"ec46ef29-6d38-44fd-a742-e748bec0ffe6\") " pod="kuadrant-system/kuadrant-operator-catalog-6lgzr" Jul 07 15:46:23.902362 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:23.902292 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6lgzr" Jul 07 15:46:24.047958 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:24.047929 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6lgzr"] Jul 07 15:46:24.050838 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:46:24.050809 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec46ef29_6d38_44fd_a742_e748bec0ffe6.slice/crio-35940ca07a248e49787d5abd5e6cec99ca1d25be9cf9a9fdafa128973f23083a WatchSource:0}: Error finding container 35940ca07a248e49787d5abd5e6cec99ca1d25be9cf9a9fdafa128973f23083a: Status 404 returned error can't find the container with id 35940ca07a248e49787d5abd5e6cec99ca1d25be9cf9a9fdafa128973f23083a Jul 07 15:46:24.284733 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:24.284702 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rgj79" event={"ID":"c4a79fe8-363d-4b53-8e81-758d9b404545","Type":"ContainerStarted","Data":"7617f2cf96a7812f50a6045666ad3b9e7f3347e0b167edf508b9793235f21da1"} Jul 07 15:46:24.285866 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:24.285840 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6lgzr" event={"ID":"ec46ef29-6d38-44fd-a742-e748bec0ffe6","Type":"ContainerStarted","Data":"35940ca07a248e49787d5abd5e6cec99ca1d25be9cf9a9fdafa128973f23083a"} Jul 07 15:46:26.298332 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:26.298285 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rgj79" event={"ID":"c4a79fe8-363d-4b53-8e81-758d9b404545","Type":"ContainerStarted","Data":"a8e86aea52970de933d30c8613163b111b0150c657412e3c399bce3f6b3d39b5"} Jul 07 15:46:26.298332 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:26.298310 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-rgj79" podUID="c4a79fe8-363d-4b53-8e81-758d9b404545" containerName="registry-server" containerID="cri-o://a8e86aea52970de933d30c8613163b111b0150c657412e3c399bce3f6b3d39b5" gracePeriod=2 Jul 07 15:46:26.299823 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:26.299795 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6lgzr" event={"ID":"ec46ef29-6d38-44fd-a742-e748bec0ffe6","Type":"ContainerStarted","Data":"89b3d2e929b045afbc5f865fdd6d0d98ffbb357d1a84c9ce6282539371e3fb03"} Jul 07 15:46:26.315166 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:26.315030 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-rgj79" podStartSLOduration=2.105667092 podStartE2EDuration="4.315010391s" totalImagesPullingTime="2.209343299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:46:26.314572014 +0000 UTC m=+370.492216474" watchObservedRunningTime="2026-07-07 15:46:26.315010391 +0000 UTC m=+370.492654868" Jul 07 15:46:26.331242 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:26.331201 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-6lgzr" podStartSLOduration=1.7049036869999998 podStartE2EDuration="3.331184645s" totalImagesPullingTime="1.626280958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:46:26.330191383 +0000 UTC m=+370.507835855" watchObservedRunningTime="2026-07-07 15:46:26.331184645 +0000 UTC m=+370.508829091" Jul 07 15:46:26.546773 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:26.546744 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rgj79" Jul 07 15:46:26.649886 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:26.649797 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwx8f\" (UniqueName: \"kubernetes.io/projected/c4a79fe8-363d-4b53-8e81-758d9b404545-kube-api-access-fwx8f\") pod \"c4a79fe8-363d-4b53-8e81-758d9b404545\" (UID: \"c4a79fe8-363d-4b53-8e81-758d9b404545\") " Jul 07 15:46:26.651971 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:26.651945 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a79fe8-363d-4b53-8e81-758d9b404545-kube-api-access-fwx8f" (OuterVolumeSpecName: "kube-api-access-fwx8f") pod "c4a79fe8-363d-4b53-8e81-758d9b404545" (UID: "c4a79fe8-363d-4b53-8e81-758d9b404545"). InnerVolumeSpecName "kube-api-access-fwx8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:46:26.750417 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:26.750382 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwx8f\" (UniqueName: \"kubernetes.io/projected/c4a79fe8-363d-4b53-8e81-758d9b404545-kube-api-access-fwx8f\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:46:27.306653 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:27.306619 2557 generic.go:356] "Generic (PLEG): container finished" podID="c4a79fe8-363d-4b53-8e81-758d9b404545" containerID="a8e86aea52970de933d30c8613163b111b0150c657412e3c399bce3f6b3d39b5" exitCode=0 Jul 07 15:46:27.307183 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:27.306663 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rgj79" Jul 07 15:46:27.307183 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:27.306702 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rgj79" event={"ID":"c4a79fe8-363d-4b53-8e81-758d9b404545","Type":"ContainerDied","Data":"a8e86aea52970de933d30c8613163b111b0150c657412e3c399bce3f6b3d39b5"} Jul 07 15:46:27.307183 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:27.306752 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rgj79" event={"ID":"c4a79fe8-363d-4b53-8e81-758d9b404545","Type":"ContainerDied","Data":"7617f2cf96a7812f50a6045666ad3b9e7f3347e0b167edf508b9793235f21da1"} Jul 07 15:46:27.307183 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:27.306781 2557 scope.go:117] "RemoveContainer" containerID="a8e86aea52970de933d30c8613163b111b0150c657412e3c399bce3f6b3d39b5" Jul 07 15:46:27.316053 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:27.316037 2557 scope.go:117] "RemoveContainer" containerID="a8e86aea52970de933d30c8613163b111b0150c657412e3c399bce3f6b3d39b5" Jul 07 15:46:27.316328 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:46:27.316310 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e86aea52970de933d30c8613163b111b0150c657412e3c399bce3f6b3d39b5\": container with ID starting with a8e86aea52970de933d30c8613163b111b0150c657412e3c399bce3f6b3d39b5 not found: ID does not exist" containerID="a8e86aea52970de933d30c8613163b111b0150c657412e3c399bce3f6b3d39b5" Jul 07 15:46:27.316397 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:27.316338 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e86aea52970de933d30c8613163b111b0150c657412e3c399bce3f6b3d39b5"} err="failed to get container status \"a8e86aea52970de933d30c8613163b111b0150c657412e3c399bce3f6b3d39b5\": rpc error: code = NotFound desc = could not find container \"a8e86aea52970de933d30c8613163b111b0150c657412e3c399bce3f6b3d39b5\": container with ID starting with a8e86aea52970de933d30c8613163b111b0150c657412e3c399bce3f6b3d39b5 not found: ID does not exist" Jul 07 15:46:27.327525 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:27.327499 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rgj79"] Jul 07 15:46:27.331628 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:27.331601 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rgj79"] Jul 07 15:46:28.350601 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:28.350567 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4a79fe8-363d-4b53-8e81-758d9b404545" path="/var/lib/kubelet/pods/c4a79fe8-363d-4b53-8e81-758d9b404545/volumes" Jul 07 15:46:33.902939 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:33.902897 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-6lgzr" Jul 07 15:46:33.903396 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:33.902969 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-6lgzr" Jul 07 15:46:33.926251 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:33.926222 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-6lgzr" Jul 07 15:46:34.364895 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:34.364865 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-6lgzr" Jul 07 15:46:38.431135 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:38.431084 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9"] Jul 07 15:46:38.431614 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:38.431423 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4a79fe8-363d-4b53-8e81-758d9b404545" containerName="registry-server" Jul 07 15:46:38.431614 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:38.431435 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4a79fe8-363d-4b53-8e81-758d9b404545" containerName="registry-server" Jul 07 15:46:38.431614 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:38.431505 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4a79fe8-363d-4b53-8e81-758d9b404545" containerName="registry-server" Jul 07 15:46:38.436089 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:38.436071 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9" Jul 07 15:46:38.438066 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:38.438046 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gnpsb\"" Jul 07 15:46:38.444372 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:38.444349 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9"] Jul 07 15:46:38.537077 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:38.537041 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/026d55bd-3c74-427f-a8c9-62b3011eb34c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9\" (UID: \"026d55bd-3c74-427f-a8c9-62b3011eb34c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9" Jul 07 15:46:38.537253 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:38.537139 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/026d55bd-3c74-427f-a8c9-62b3011eb34c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9\" (UID: \"026d55bd-3c74-427f-a8c9-62b3011eb34c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9" Jul 07 15:46:38.537253 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:38.537166 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qkn4\" (UniqueName: \"kubernetes.io/projected/026d55bd-3c74-427f-a8c9-62b3011eb34c-kube-api-access-9qkn4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9\" (UID: \"026d55bd-3c74-427f-a8c9-62b3011eb34c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9" Jul 07 15:46:38.637615 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:38.637574 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/026d55bd-3c74-427f-a8c9-62b3011eb34c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9\" (UID: \"026d55bd-3c74-427f-a8c9-62b3011eb34c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9" Jul 07 15:46:38.637782 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:38.637626 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qkn4\" (UniqueName: \"kubernetes.io/projected/026d55bd-3c74-427f-a8c9-62b3011eb34c-kube-api-access-9qkn4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9\" (UID: \"026d55bd-3c74-427f-a8c9-62b3011eb34c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9" Jul 07 15:46:38.637782 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:38.637681 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/026d55bd-3c74-427f-a8c9-62b3011eb34c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9\" (UID: \"026d55bd-3c74-427f-a8c9-62b3011eb34c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9" Jul 07 15:46:38.638045 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:38.638029 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/026d55bd-3c74-427f-a8c9-62b3011eb34c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9\" (UID: \"026d55bd-3c74-427f-a8c9-62b3011eb34c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9" Jul 07 15:46:38.638080 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:38.638029 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/026d55bd-3c74-427f-a8c9-62b3011eb34c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9\" (UID: \"026d55bd-3c74-427f-a8c9-62b3011eb34c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9" Jul 07 15:46:38.648437 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:38.648406 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qkn4\" (UniqueName: \"kubernetes.io/projected/026d55bd-3c74-427f-a8c9-62b3011eb34c-kube-api-access-9qkn4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9\" (UID: \"026d55bd-3c74-427f-a8c9-62b3011eb34c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9" Jul 07 15:46:38.745470 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:38.745378 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9" Jul 07 15:46:38.883436 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:38.883412 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9"] Jul 07 15:46:38.885362 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:46:38.885329 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026d55bd_3c74_427f_a8c9_62b3011eb34c.slice/crio-ecbaffa62aaa0423f0b9663132b877866e87e7bde3f6c2ff7c0da20ba90c97fc WatchSource:0}: Error finding container ecbaffa62aaa0423f0b9663132b877866e87e7bde3f6c2ff7c0da20ba90c97fc: Status 404 returned error can't find the container with id ecbaffa62aaa0423f0b9663132b877866e87e7bde3f6c2ff7c0da20ba90c97fc Jul 07 15:46:39.032626 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.032547 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr"] Jul 07 15:46:39.036042 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.036026 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr" Jul 07 15:46:39.046952 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.046930 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr"] Jul 07 15:46:39.142458 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.142410 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p66gd\" (UniqueName: \"kubernetes.io/projected/bae8fc14-9c5a-4ed3-b35a-0406be00443e-kube-api-access-p66gd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr\" (UID: \"bae8fc14-9c5a-4ed3-b35a-0406be00443e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr" Jul 07 15:46:39.142636 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.142499 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bae8fc14-9c5a-4ed3-b35a-0406be00443e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr\" (UID: \"bae8fc14-9c5a-4ed3-b35a-0406be00443e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr" Jul 07 15:46:39.142636 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.142518 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bae8fc14-9c5a-4ed3-b35a-0406be00443e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr\" (UID: \"bae8fc14-9c5a-4ed3-b35a-0406be00443e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr" Jul 07 15:46:39.243346 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.243310 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p66gd\" (UniqueName: \"kubernetes.io/projected/bae8fc14-9c5a-4ed3-b35a-0406be00443e-kube-api-access-p66gd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr\" (UID: \"bae8fc14-9c5a-4ed3-b35a-0406be00443e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr" Jul 07 15:46:39.243484 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.243367 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bae8fc14-9c5a-4ed3-b35a-0406be00443e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr\" (UID: \"bae8fc14-9c5a-4ed3-b35a-0406be00443e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr" Jul 07 15:46:39.243484 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.243385 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bae8fc14-9c5a-4ed3-b35a-0406be00443e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr\" (UID: \"bae8fc14-9c5a-4ed3-b35a-0406be00443e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr" Jul 07 15:46:39.243761 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.243740 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bae8fc14-9c5a-4ed3-b35a-0406be00443e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr\" (UID: \"bae8fc14-9c5a-4ed3-b35a-0406be00443e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr" Jul 07 15:46:39.243802 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.243752 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bae8fc14-9c5a-4ed3-b35a-0406be00443e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr\" (UID: \"bae8fc14-9c5a-4ed3-b35a-0406be00443e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr" Jul 07 15:46:39.251040 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.251020 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p66gd\" (UniqueName: \"kubernetes.io/projected/bae8fc14-9c5a-4ed3-b35a-0406be00443e-kube-api-access-p66gd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr\" (UID: \"bae8fc14-9c5a-4ed3-b35a-0406be00443e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr" Jul 07 15:46:39.366879 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.366851 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr" Jul 07 15:46:39.368708 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.368679 2557 generic.go:356] "Generic (PLEG): container finished" podID="026d55bd-3c74-427f-a8c9-62b3011eb34c" containerID="c0349ccbbc87f508519b1ebe357053ad39b909293dc0e2e0842c385923e37303" exitCode=0 Jul 07 15:46:39.368841 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.368761 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9" event={"ID":"026d55bd-3c74-427f-a8c9-62b3011eb34c","Type":"ContainerDied","Data":"c0349ccbbc87f508519b1ebe357053ad39b909293dc0e2e0842c385923e37303"} Jul 07 15:46:39.368841 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.368805 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9" event={"ID":"026d55bd-3c74-427f-a8c9-62b3011eb34c","Type":"ContainerStarted","Data":"ecbaffa62aaa0423f0b9663132b877866e87e7bde3f6c2ff7c0da20ba90c97fc"} Jul 07 15:46:39.431493 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.431463 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc"] Jul 07 15:46:39.442003 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.441660 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc" Jul 07 15:46:39.444926 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.444894 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc"] Jul 07 15:46:39.513033 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.513009 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr"] Jul 07 15:46:39.514837 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:46:39.514804 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae8fc14_9c5a_4ed3_b35a_0406be00443e.slice/crio-f2b9d8df3a0817a87089779fb1968fbed688ce1dd61bdf563c9c9d44113cb700 WatchSource:0}: Error finding container f2b9d8df3a0817a87089779fb1968fbed688ce1dd61bdf563c9c9d44113cb700: Status 404 returned error can't find the container with id f2b9d8df3a0817a87089779fb1968fbed688ce1dd61bdf563c9c9d44113cb700 Jul 07 15:46:39.546340 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.546317 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f53c3f49-cca3-4660-96d2-a3a32b5a39c6-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc\" (UID: \"f53c3f49-cca3-4660-96d2-a3a32b5a39c6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc" Jul 07 15:46:39.546431 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.546362 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f53c3f49-cca3-4660-96d2-a3a32b5a39c6-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc\" (UID: \"f53c3f49-cca3-4660-96d2-a3a32b5a39c6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc" Jul 07 15:46:39.546431 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.546389 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tgvb\" (UniqueName: \"kubernetes.io/projected/f53c3f49-cca3-4660-96d2-a3a32b5a39c6-kube-api-access-9tgvb\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc\" (UID: \"f53c3f49-cca3-4660-96d2-a3a32b5a39c6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc" Jul 07 15:46:39.647707 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.647675 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f53c3f49-cca3-4660-96d2-a3a32b5a39c6-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc\" (UID: \"f53c3f49-cca3-4660-96d2-a3a32b5a39c6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc" Jul 07 15:46:39.647860 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.647722 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tgvb\" (UniqueName: \"kubernetes.io/projected/f53c3f49-cca3-4660-96d2-a3a32b5a39c6-kube-api-access-9tgvb\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc\" (UID: \"f53c3f49-cca3-4660-96d2-a3a32b5a39c6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc" Jul 07 15:46:39.647860 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.647777 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f53c3f49-cca3-4660-96d2-a3a32b5a39c6-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc\" (UID: \"f53c3f49-cca3-4660-96d2-a3a32b5a39c6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc" Jul 07 15:46:39.648096 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.648074 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f53c3f49-cca3-4660-96d2-a3a32b5a39c6-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc\" (UID: \"f53c3f49-cca3-4660-96d2-a3a32b5a39c6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc" Jul 07 15:46:39.648204 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.648145 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f53c3f49-cca3-4660-96d2-a3a32b5a39c6-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc\" (UID: \"f53c3f49-cca3-4660-96d2-a3a32b5a39c6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc" Jul 07 15:46:39.655856 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.655837 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tgvb\" (UniqueName: \"kubernetes.io/projected/f53c3f49-cca3-4660-96d2-a3a32b5a39c6-kube-api-access-9tgvb\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc\" (UID: \"f53c3f49-cca3-4660-96d2-a3a32b5a39c6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc" Jul 07 15:46:39.753341 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.753299 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc" Jul 07 15:46:39.832562 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.832525 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4"] Jul 07 15:46:39.837942 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.837919 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4" Jul 07 15:46:39.849598 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.849554 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/faef017c-9227-481f-b21d-a8b7c9a214c4-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4\" (UID: \"faef017c-9227-481f-b21d-a8b7c9a214c4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4" Jul 07 15:46:39.849598 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.849579 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4"] Jul 07 15:46:39.850313 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.849661 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw54g\" (UniqueName: \"kubernetes.io/projected/faef017c-9227-481f-b21d-a8b7c9a214c4-kube-api-access-kw54g\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4\" (UID: \"faef017c-9227-481f-b21d-a8b7c9a214c4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4" Jul 07 15:46:39.850313 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.849715 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/faef017c-9227-481f-b21d-a8b7c9a214c4-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4\" (UID: \"faef017c-9227-481f-b21d-a8b7c9a214c4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4" Jul 07 15:46:39.897919 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.897882 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc"] Jul 07 15:46:39.900185 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:46:39.900161 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53c3f49_cca3_4660_96d2_a3a32b5a39c6.slice/crio-fc0abca5822c4bef5c2cb9e22ff45a70a0521faeaae94e82b46559c0be92a3ff WatchSource:0}: Error finding container fc0abca5822c4bef5c2cb9e22ff45a70a0521faeaae94e82b46559c0be92a3ff: Status 404 returned error can't find the container with id fc0abca5822c4bef5c2cb9e22ff45a70a0521faeaae94e82b46559c0be92a3ff Jul 07 15:46:39.950444 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.950417 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/faef017c-9227-481f-b21d-a8b7c9a214c4-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4\" (UID: \"faef017c-9227-481f-b21d-a8b7c9a214c4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4" Jul 07 15:46:39.950536 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.950458 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw54g\" (UniqueName: \"kubernetes.io/projected/faef017c-9227-481f-b21d-a8b7c9a214c4-kube-api-access-kw54g\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4\" (UID: \"faef017c-9227-481f-b21d-a8b7c9a214c4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4" Jul 07 15:46:39.950536 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.950477 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/faef017c-9227-481f-b21d-a8b7c9a214c4-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4\" (UID: \"faef017c-9227-481f-b21d-a8b7c9a214c4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4" Jul 07 15:46:39.950792 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.950776 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/faef017c-9227-481f-b21d-a8b7c9a214c4-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4\" (UID: \"faef017c-9227-481f-b21d-a8b7c9a214c4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4" Jul 07 15:46:39.950874 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.950850 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/faef017c-9227-481f-b21d-a8b7c9a214c4-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4\" (UID: \"faef017c-9227-481f-b21d-a8b7c9a214c4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4" Jul 07 15:46:39.958775 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:39.958747 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw54g\" (UniqueName: \"kubernetes.io/projected/faef017c-9227-481f-b21d-a8b7c9a214c4-kube-api-access-kw54g\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4\" (UID: \"faef017c-9227-481f-b21d-a8b7c9a214c4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4" Jul 07 15:46:40.150438 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:40.150363 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4" Jul 07 15:46:40.324848 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:40.324826 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4"] Jul 07 15:46:40.352921 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:46:40.352894 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaef017c_9227_481f_b21d_a8b7c9a214c4.slice/crio-dc0e9eb267037001dade0a70e134be4e6287499007507e11529a022b89ed008c WatchSource:0}: Error finding container dc0e9eb267037001dade0a70e134be4e6287499007507e11529a022b89ed008c: Status 404 returned error can't find the container with id dc0e9eb267037001dade0a70e134be4e6287499007507e11529a022b89ed008c Jul 07 15:46:40.376435 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:40.376408 2557 generic.go:356] "Generic (PLEG): container finished" podID="026d55bd-3c74-427f-a8c9-62b3011eb34c" containerID="115a93d7ee32bba51c14e0eaeb6351aec8b549b23f68b38926c83b6bc89069c3" exitCode=0 Jul 07 15:46:40.376512 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:40.376487 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9" event={"ID":"026d55bd-3c74-427f-a8c9-62b3011eb34c","Type":"ContainerDied","Data":"115a93d7ee32bba51c14e0eaeb6351aec8b549b23f68b38926c83b6bc89069c3"} Jul 07 15:46:40.377659 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:40.377623 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4" event={"ID":"faef017c-9227-481f-b21d-a8b7c9a214c4","Type":"ContainerStarted","Data":"dc0e9eb267037001dade0a70e134be4e6287499007507e11529a022b89ed008c"} Jul 07 15:46:40.379099 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:40.379077 2557 generic.go:356] "Generic (PLEG): container finished" podID="bae8fc14-9c5a-4ed3-b35a-0406be00443e" containerID="fa76a9c045310b35290238ca9eaf8b00a1245242c9ef90f6a464edb3462b6bcc" exitCode=0 Jul 07 15:46:40.379214 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:40.379140 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr" event={"ID":"bae8fc14-9c5a-4ed3-b35a-0406be00443e","Type":"ContainerDied","Data":"fa76a9c045310b35290238ca9eaf8b00a1245242c9ef90f6a464edb3462b6bcc"} Jul 07 15:46:40.379214 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:40.379176 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr" event={"ID":"bae8fc14-9c5a-4ed3-b35a-0406be00443e","Type":"ContainerStarted","Data":"f2b9d8df3a0817a87089779fb1968fbed688ce1dd61bdf563c9c9d44113cb700"} Jul 07 15:46:40.380765 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:40.380741 2557 generic.go:356] "Generic (PLEG): container finished" podID="f53c3f49-cca3-4660-96d2-a3a32b5a39c6" containerID="2cfc47c37e8fec37eb4675fc50b7afda3469f4f7ae05a00def2f95948ccb894f" exitCode=0 Jul 07 15:46:40.380849 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:40.380790 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc" event={"ID":"f53c3f49-cca3-4660-96d2-a3a32b5a39c6","Type":"ContainerDied","Data":"2cfc47c37e8fec37eb4675fc50b7afda3469f4f7ae05a00def2f95948ccb894f"} Jul 07 15:46:40.380849 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:40.380831 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc" event={"ID":"f53c3f49-cca3-4660-96d2-a3a32b5a39c6","Type":"ContainerStarted","Data":"fc0abca5822c4bef5c2cb9e22ff45a70a0521faeaae94e82b46559c0be92a3ff"} Jul 07 15:46:41.388209 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:41.388178 2557 generic.go:356] "Generic (PLEG): container finished" podID="026d55bd-3c74-427f-a8c9-62b3011eb34c" containerID="7ed414caac09685aa88a277677dc6ca24951aedec228e88ad17abffb320254a5" exitCode=0 Jul 07 15:46:41.388530 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:41.388259 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9" event={"ID":"026d55bd-3c74-427f-a8c9-62b3011eb34c","Type":"ContainerDied","Data":"7ed414caac09685aa88a277677dc6ca24951aedec228e88ad17abffb320254a5"} Jul 07 15:46:41.389773 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:41.389751 2557 generic.go:356] "Generic (PLEG): container finished" podID="faef017c-9227-481f-b21d-a8b7c9a214c4" containerID="ed0c69f1ee68d7f20452b8933a540866bd3d25c035c64a7f03955eb716034419" exitCode=0 Jul 07 15:46:41.389875 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:41.389797 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4" event={"ID":"faef017c-9227-481f-b21d-a8b7c9a214c4","Type":"ContainerDied","Data":"ed0c69f1ee68d7f20452b8933a540866bd3d25c035c64a7f03955eb716034419"} Jul 07 15:46:42.395919 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:42.395837 2557 generic.go:356] "Generic (PLEG): container finished" podID="faef017c-9227-481f-b21d-a8b7c9a214c4" containerID="67f859394785a41ee0bc3c57081947b2962086f9ca49859fed9b9513c47f0038" exitCode=0 Jul 07 15:46:42.396336 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:42.395928 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4" event={"ID":"faef017c-9227-481f-b21d-a8b7c9a214c4","Type":"ContainerDied","Data":"67f859394785a41ee0bc3c57081947b2962086f9ca49859fed9b9513c47f0038"} Jul 07 15:46:42.397883 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:42.397856 2557 generic.go:356] "Generic (PLEG): container finished" podID="bae8fc14-9c5a-4ed3-b35a-0406be00443e" containerID="5ea0d0d083c998e39666c8ff43c4b42e877537f6f69d9ed74080ec5203ff231f" exitCode=0 Jul 07 15:46:42.397975 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:42.397931 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr" event={"ID":"bae8fc14-9c5a-4ed3-b35a-0406be00443e","Type":"ContainerDied","Data":"5ea0d0d083c998e39666c8ff43c4b42e877537f6f69d9ed74080ec5203ff231f"} Jul 07 15:46:42.399871 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:42.399847 2557 generic.go:356] "Generic (PLEG): container finished" podID="f53c3f49-cca3-4660-96d2-a3a32b5a39c6" containerID="179a97e46fa86f02dd9b008ad98e73ea42e08fbca47bbb5198705d0dde45e9e5" exitCode=0 Jul 07 15:46:42.399955 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:42.399920 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc" event={"ID":"f53c3f49-cca3-4660-96d2-a3a32b5a39c6","Type":"ContainerDied","Data":"179a97e46fa86f02dd9b008ad98e73ea42e08fbca47bbb5198705d0dde45e9e5"} Jul 07 15:46:42.551197 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:42.551167 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9" Jul 07 15:46:42.574322 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:42.574237 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/026d55bd-3c74-427f-a8c9-62b3011eb34c-bundle\") pod \"026d55bd-3c74-427f-a8c9-62b3011eb34c\" (UID: \"026d55bd-3c74-427f-a8c9-62b3011eb34c\") " Jul 07 15:46:42.574553 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:42.574362 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/026d55bd-3c74-427f-a8c9-62b3011eb34c-util\") pod \"026d55bd-3c74-427f-a8c9-62b3011eb34c\" (UID: \"026d55bd-3c74-427f-a8c9-62b3011eb34c\") " Jul 07 15:46:42.574553 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:42.574421 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qkn4\" (UniqueName: \"kubernetes.io/projected/026d55bd-3c74-427f-a8c9-62b3011eb34c-kube-api-access-9qkn4\") pod \"026d55bd-3c74-427f-a8c9-62b3011eb34c\" (UID: \"026d55bd-3c74-427f-a8c9-62b3011eb34c\") " Jul 07 15:46:42.574792 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:42.574768 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026d55bd-3c74-427f-a8c9-62b3011eb34c-bundle" (OuterVolumeSpecName: "bundle") pod "026d55bd-3c74-427f-a8c9-62b3011eb34c" (UID: "026d55bd-3c74-427f-a8c9-62b3011eb34c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:46:42.576759 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:42.576732 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026d55bd-3c74-427f-a8c9-62b3011eb34c-kube-api-access-9qkn4" (OuterVolumeSpecName: "kube-api-access-9qkn4") pod "026d55bd-3c74-427f-a8c9-62b3011eb34c" (UID: "026d55bd-3c74-427f-a8c9-62b3011eb34c"). InnerVolumeSpecName "kube-api-access-9qkn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:46:42.582203 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:42.582147 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026d55bd-3c74-427f-a8c9-62b3011eb34c-util" (OuterVolumeSpecName: "util") pod "026d55bd-3c74-427f-a8c9-62b3011eb34c" (UID: "026d55bd-3c74-427f-a8c9-62b3011eb34c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:46:42.676033 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:42.675973 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/026d55bd-3c74-427f-a8c9-62b3011eb34c-bundle\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:46:42.676033 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:42.675999 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/026d55bd-3c74-427f-a8c9-62b3011eb34c-util\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:46:42.676033 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:42.676010 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qkn4\" (UniqueName: \"kubernetes.io/projected/026d55bd-3c74-427f-a8c9-62b3011eb34c-kube-api-access-9qkn4\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:46:43.407506 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:43.407420 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9" Jul 07 15:46:43.407506 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:43.407420 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9" event={"ID":"026d55bd-3c74-427f-a8c9-62b3011eb34c","Type":"ContainerDied","Data":"ecbaffa62aaa0423f0b9663132b877866e87e7bde3f6c2ff7c0da20ba90c97fc"} Jul 07 15:46:43.407961 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:43.407540 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecbaffa62aaa0423f0b9663132b877866e87e7bde3f6c2ff7c0da20ba90c97fc" Jul 07 15:46:43.409604 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:43.409572 2557 generic.go:356] "Generic (PLEG): container finished" podID="faef017c-9227-481f-b21d-a8b7c9a214c4" containerID="7c3570b599f0e7bff92456b72a938c7cd6f721e4848421ed836f33582cb29a27" exitCode=0 Jul 07 15:46:43.409743 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:43.409649 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4" event={"ID":"faef017c-9227-481f-b21d-a8b7c9a214c4","Type":"ContainerDied","Data":"7c3570b599f0e7bff92456b72a938c7cd6f721e4848421ed836f33582cb29a27"} Jul 07 15:46:43.411691 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:43.411667 2557 generic.go:356] "Generic (PLEG): container finished" podID="bae8fc14-9c5a-4ed3-b35a-0406be00443e" containerID="2483767e712b396eccc28a9e09637cf2ac5b6a4a919ce720c0def9683890e423" exitCode=0 Jul 07 15:46:43.411769 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:43.411690 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr" event={"ID":"bae8fc14-9c5a-4ed3-b35a-0406be00443e","Type":"ContainerDied","Data":"2483767e712b396eccc28a9e09637cf2ac5b6a4a919ce720c0def9683890e423"} Jul 07 15:46:43.414068 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:43.414048 2557 generic.go:356] "Generic (PLEG): container finished" podID="f53c3f49-cca3-4660-96d2-a3a32b5a39c6" containerID="df83dbccabe5e36541d89e034d8a1fe15280caf926c0fd581da2b5d4e98c775b" exitCode=0 Jul 07 15:46:43.414152 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:43.414082 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc" event={"ID":"f53c3f49-cca3-4660-96d2-a3a32b5a39c6","Type":"ContainerDied","Data":"df83dbccabe5e36541d89e034d8a1fe15280caf926c0fd581da2b5d4e98c775b"} Jul 07 15:46:44.579748 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.579724 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr" Jul 07 15:46:44.622296 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.622269 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc" Jul 07 15:46:44.625336 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.625316 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4" Jul 07 15:46:44.695553 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.695459 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kw54g\" (UniqueName: \"kubernetes.io/projected/faef017c-9227-481f-b21d-a8b7c9a214c4-kube-api-access-kw54g\") pod \"faef017c-9227-481f-b21d-a8b7c9a214c4\" (UID: \"faef017c-9227-481f-b21d-a8b7c9a214c4\") " Jul 07 15:46:44.695553 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.695502 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bae8fc14-9c5a-4ed3-b35a-0406be00443e-util\") pod \"bae8fc14-9c5a-4ed3-b35a-0406be00443e\" (UID: \"bae8fc14-9c5a-4ed3-b35a-0406be00443e\") " Jul 07 15:46:44.695553 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.695523 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/faef017c-9227-481f-b21d-a8b7c9a214c4-bundle\") pod \"faef017c-9227-481f-b21d-a8b7c9a214c4\" (UID: \"faef017c-9227-481f-b21d-a8b7c9a214c4\") " Jul 07 15:46:44.695553 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.695546 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f53c3f49-cca3-4660-96d2-a3a32b5a39c6-bundle\") pod \"f53c3f49-cca3-4660-96d2-a3a32b5a39c6\" (UID: \"f53c3f49-cca3-4660-96d2-a3a32b5a39c6\") " Jul 07 15:46:44.695881 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.695579 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f53c3f49-cca3-4660-96d2-a3a32b5a39c6-util\") pod \"f53c3f49-cca3-4660-96d2-a3a32b5a39c6\" (UID: \"f53c3f49-cca3-4660-96d2-a3a32b5a39c6\") " Jul 07 15:46:44.695881 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.695613 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tgvb\" (UniqueName: \"kubernetes.io/projected/f53c3f49-cca3-4660-96d2-a3a32b5a39c6-kube-api-access-9tgvb\") pod \"f53c3f49-cca3-4660-96d2-a3a32b5a39c6\" (UID: \"f53c3f49-cca3-4660-96d2-a3a32b5a39c6\") " Jul 07 15:46:44.695881 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.695628 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/faef017c-9227-481f-b21d-a8b7c9a214c4-util\") pod \"faef017c-9227-481f-b21d-a8b7c9a214c4\" (UID: \"faef017c-9227-481f-b21d-a8b7c9a214c4\") " Jul 07 15:46:44.695881 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.695663 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p66gd\" (UniqueName: \"kubernetes.io/projected/bae8fc14-9c5a-4ed3-b35a-0406be00443e-kube-api-access-p66gd\") pod \"bae8fc14-9c5a-4ed3-b35a-0406be00443e\" (UID: \"bae8fc14-9c5a-4ed3-b35a-0406be00443e\") " Jul 07 15:46:44.695881 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.695706 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bae8fc14-9c5a-4ed3-b35a-0406be00443e-bundle\") pod \"bae8fc14-9c5a-4ed3-b35a-0406be00443e\" (UID: \"bae8fc14-9c5a-4ed3-b35a-0406be00443e\") " Jul 07 15:46:44.696331 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.696304 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faef017c-9227-481f-b21d-a8b7c9a214c4-bundle" (OuterVolumeSpecName: "bundle") pod "faef017c-9227-481f-b21d-a8b7c9a214c4" (UID: "faef017c-9227-481f-b21d-a8b7c9a214c4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:46:44.696425 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.696404 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae8fc14-9c5a-4ed3-b35a-0406be00443e-bundle" (OuterVolumeSpecName: "bundle") pod "bae8fc14-9c5a-4ed3-b35a-0406be00443e" (UID: "bae8fc14-9c5a-4ed3-b35a-0406be00443e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:46:44.696927 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.696888 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53c3f49-cca3-4660-96d2-a3a32b5a39c6-bundle" (OuterVolumeSpecName: "bundle") pod "f53c3f49-cca3-4660-96d2-a3a32b5a39c6" (UID: "f53c3f49-cca3-4660-96d2-a3a32b5a39c6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:46:44.698054 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.698025 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53c3f49-cca3-4660-96d2-a3a32b5a39c6-kube-api-access-9tgvb" (OuterVolumeSpecName: "kube-api-access-9tgvb") pod "f53c3f49-cca3-4660-96d2-a3a32b5a39c6" (UID: "f53c3f49-cca3-4660-96d2-a3a32b5a39c6"). InnerVolumeSpecName "kube-api-access-9tgvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:46:44.698440 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.698419 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faef017c-9227-481f-b21d-a8b7c9a214c4-kube-api-access-kw54g" (OuterVolumeSpecName: "kube-api-access-kw54g") pod "faef017c-9227-481f-b21d-a8b7c9a214c4" (UID: "faef017c-9227-481f-b21d-a8b7c9a214c4"). InnerVolumeSpecName "kube-api-access-kw54g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:46:44.698640 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.698620 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae8fc14-9c5a-4ed3-b35a-0406be00443e-kube-api-access-p66gd" (OuterVolumeSpecName: "kube-api-access-p66gd") pod "bae8fc14-9c5a-4ed3-b35a-0406be00443e" (UID: "bae8fc14-9c5a-4ed3-b35a-0406be00443e"). InnerVolumeSpecName "kube-api-access-p66gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:46:44.702068 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.702046 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae8fc14-9c5a-4ed3-b35a-0406be00443e-util" (OuterVolumeSpecName: "util") pod "bae8fc14-9c5a-4ed3-b35a-0406be00443e" (UID: "bae8fc14-9c5a-4ed3-b35a-0406be00443e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:46:44.704422 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.704381 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53c3f49-cca3-4660-96d2-a3a32b5a39c6-util" (OuterVolumeSpecName: "util") pod "f53c3f49-cca3-4660-96d2-a3a32b5a39c6" (UID: "f53c3f49-cca3-4660-96d2-a3a32b5a39c6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:46:44.704813 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.704795 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faef017c-9227-481f-b21d-a8b7c9a214c4-util" (OuterVolumeSpecName: "util") pod "faef017c-9227-481f-b21d-a8b7c9a214c4" (UID: "faef017c-9227-481f-b21d-a8b7c9a214c4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:46:44.797416 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.797374 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p66gd\" (UniqueName: \"kubernetes.io/projected/bae8fc14-9c5a-4ed3-b35a-0406be00443e-kube-api-access-p66gd\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:46:44.797416 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.797410 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bae8fc14-9c5a-4ed3-b35a-0406be00443e-bundle\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:46:44.797416 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.797419 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw54g\" (UniqueName: \"kubernetes.io/projected/faef017c-9227-481f-b21d-a8b7c9a214c4-kube-api-access-kw54g\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:46:44.797416 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.797427 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bae8fc14-9c5a-4ed3-b35a-0406be00443e-util\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:46:44.797680 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.797436 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/faef017c-9227-481f-b21d-a8b7c9a214c4-bundle\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:46:44.797680 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.797443 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f53c3f49-cca3-4660-96d2-a3a32b5a39c6-bundle\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:46:44.797680 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.797450 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f53c3f49-cca3-4660-96d2-a3a32b5a39c6-util\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:46:44.797680 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.797458 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tgvb\" (UniqueName: \"kubernetes.io/projected/f53c3f49-cca3-4660-96d2-a3a32b5a39c6-kube-api-access-9tgvb\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:46:44.797680 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:44.797474 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/faef017c-9227-481f-b21d-a8b7c9a214c4-util\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:46:45.426513 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:45.426480 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4" Jul 07 15:46:45.426513 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:45.426491 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4" event={"ID":"faef017c-9227-481f-b21d-a8b7c9a214c4","Type":"ContainerDied","Data":"dc0e9eb267037001dade0a70e134be4e6287499007507e11529a022b89ed008c"} Jul 07 15:46:45.426728 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:45.426530 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc0e9eb267037001dade0a70e134be4e6287499007507e11529a022b89ed008c" Jul 07 15:46:45.428326 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:45.428307 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr" Jul 07 15:46:45.428326 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:45.428314 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr" event={"ID":"bae8fc14-9c5a-4ed3-b35a-0406be00443e","Type":"ContainerDied","Data":"f2b9d8df3a0817a87089779fb1968fbed688ce1dd61bdf563c9c9d44113cb700"} Jul 07 15:46:45.428496 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:45.428346 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2b9d8df3a0817a87089779fb1968fbed688ce1dd61bdf563c9c9d44113cb700" Jul 07 15:46:45.430554 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:45.430536 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc" Jul 07 15:46:45.430647 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:45.430552 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc" event={"ID":"f53c3f49-cca3-4660-96d2-a3a32b5a39c6","Type":"ContainerDied","Data":"fc0abca5822c4bef5c2cb9e22ff45a70a0521faeaae94e82b46559c0be92a3ff"} Jul 07 15:46:45.430647 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:45.430579 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc0abca5822c4bef5c2cb9e22ff45a70a0521faeaae94e82b46559c0be92a3ff" Jul 07 15:46:54.373586 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.373554 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z"] Jul 07 15:46:54.373951 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.373892 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bae8fc14-9c5a-4ed3-b35a-0406be00443e" containerName="pull" Jul 07 15:46:54.373951 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.373903 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bae8fc14-9c5a-4ed3-b35a-0406be00443e" containerName="pull" Jul 07 15:46:54.373951 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.373915 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="026d55bd-3c74-427f-a8c9-62b3011eb34c" containerName="util" Jul 07 15:46:54.373951 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.373920 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="026d55bd-3c74-427f-a8c9-62b3011eb34c" containerName="util" Jul 07 15:46:54.373951 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.373928 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f53c3f49-cca3-4660-96d2-a3a32b5a39c6" containerName="util" Jul 07 15:46:54.373951 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.373933 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f53c3f49-cca3-4660-96d2-a3a32b5a39c6" containerName="util" Jul 07 15:46:54.373951 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.373942 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="026d55bd-3c74-427f-a8c9-62b3011eb34c" containerName="extract" Jul 07 15:46:54.373951 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.373949 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="026d55bd-3c74-427f-a8c9-62b3011eb34c" containerName="extract" Jul 07 15:46:54.373951 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.373954 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f53c3f49-cca3-4660-96d2-a3a32b5a39c6" containerName="extract" Jul 07 15:46:54.374259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.373960 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f53c3f49-cca3-4660-96d2-a3a32b5a39c6" containerName="extract" Jul 07 15:46:54.374259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.373967 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bae8fc14-9c5a-4ed3-b35a-0406be00443e" containerName="util" Jul 07 15:46:54.374259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.373972 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bae8fc14-9c5a-4ed3-b35a-0406be00443e" containerName="util" Jul 07 15:46:54.374259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.373979 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bae8fc14-9c5a-4ed3-b35a-0406be00443e" containerName="extract" Jul 07 15:46:54.374259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.373984 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bae8fc14-9c5a-4ed3-b35a-0406be00443e" containerName="extract" Jul 07 15:46:54.374259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.373990 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faef017c-9227-481f-b21d-a8b7c9a214c4" containerName="extract" Jul 07 15:46:54.374259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.373995 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="faef017c-9227-481f-b21d-a8b7c9a214c4" containerName="extract" Jul 07 15:46:54.374259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.374003 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f53c3f49-cca3-4660-96d2-a3a32b5a39c6" containerName="pull" Jul 07 15:46:54.374259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.374008 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f53c3f49-cca3-4660-96d2-a3a32b5a39c6" containerName="pull" Jul 07 15:46:54.374259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.374015 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="026d55bd-3c74-427f-a8c9-62b3011eb34c" containerName="pull" Jul 07 15:46:54.374259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.374019 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="026d55bd-3c74-427f-a8c9-62b3011eb34c" containerName="pull" Jul 07 15:46:54.374259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.374026 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faef017c-9227-481f-b21d-a8b7c9a214c4" containerName="util" Jul 07 15:46:54.374259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.374031 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="faef017c-9227-481f-b21d-a8b7c9a214c4" containerName="util" Jul 07 15:46:54.374259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.374036 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faef017c-9227-481f-b21d-a8b7c9a214c4" containerName="pull" Jul 07 15:46:54.374259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.374040 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="faef017c-9227-481f-b21d-a8b7c9a214c4" containerName="pull" Jul 07 15:46:54.374259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.374091 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f53c3f49-cca3-4660-96d2-a3a32b5a39c6" containerName="extract" Jul 07 15:46:54.374259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.374099 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="026d55bd-3c74-427f-a8c9-62b3011eb34c" containerName="extract" Jul 07 15:46:54.374259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.374105 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="faef017c-9227-481f-b21d-a8b7c9a214c4" containerName="extract" Jul 07 15:46:54.374259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.374132 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bae8fc14-9c5a-4ed3-b35a-0406be00443e" containerName="extract" Jul 07 15:46:54.378545 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.378525 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z" Jul 07 15:46:54.381565 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.381546 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-wrn85\"" Jul 07 15:46:54.390411 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.390387 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z"] Jul 07 15:46:54.482511 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.482477 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx9mt\" (UniqueName: \"kubernetes.io/projected/b1bbbb98-c897-409a-8667-14c39593c63e-kube-api-access-xx9mt\") pod \"limitador-operator-controller-manager-859d7b78bd-q7l7z\" (UID: \"b1bbbb98-c897-409a-8667-14c39593c63e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z" Jul 07 15:46:54.582968 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.582932 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx9mt\" (UniqueName: \"kubernetes.io/projected/b1bbbb98-c897-409a-8667-14c39593c63e-kube-api-access-xx9mt\") pod \"limitador-operator-controller-manager-859d7b78bd-q7l7z\" (UID: \"b1bbbb98-c897-409a-8667-14c39593c63e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z" Jul 07 15:46:54.593224 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.593195 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx9mt\" (UniqueName: \"kubernetes.io/projected/b1bbbb98-c897-409a-8667-14c39593c63e-kube-api-access-xx9mt\") pod \"limitador-operator-controller-manager-859d7b78bd-q7l7z\" (UID: \"b1bbbb98-c897-409a-8667-14c39593c63e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z" Jul 07 15:46:54.688590 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.688506 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z" Jul 07 15:46:54.831289 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:54.831263 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z"] Jul 07 15:46:54.833536 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:46:54.833508 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1bbbb98_c897_409a_8667_14c39593c63e.slice/crio-eab9271cfc1e24be7d47977e7181eb80721df03e8ba72b381c50e0411089bdf4 WatchSource:0}: Error finding container eab9271cfc1e24be7d47977e7181eb80721df03e8ba72b381c50e0411089bdf4: Status 404 returned error can't find the container with id eab9271cfc1e24be7d47977e7181eb80721df03e8ba72b381c50e0411089bdf4 Jul 07 15:46:55.488739 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:55.488706 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z" event={"ID":"b1bbbb98-c897-409a-8667-14c39593c63e","Type":"ContainerStarted","Data":"eab9271cfc1e24be7d47977e7181eb80721df03e8ba72b381c50e0411089bdf4"} Jul 07 15:46:57.500734 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:57.500695 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z" event={"ID":"b1bbbb98-c897-409a-8667-14c39593c63e","Type":"ContainerStarted","Data":"18e7d4396231c4df11f3c77ea19e838c230847abb261cfcc68f555dd15534a62"} Jul 07 15:46:57.501162 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:57.500781 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z" Jul 07 15:46:57.520747 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:57.520694 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z" podStartSLOduration=1.817486582 podStartE2EDuration="3.520678432s" totalImagesPullingTime="1.70319185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:46:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:46:57.518417754 +0000 UTC m=+401.696062229" watchObservedRunningTime="2026-07-07 15:46:57.520678432 +0000 UTC m=+401.698322881" Jul 07 15:46:57.846518 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:57.846484 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5fhk"] Jul 07 15:46:57.854316 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:57.854288 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5fhk" Jul 07 15:46:57.856689 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:57.856663 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-ssksc\"" Jul 07 15:46:57.857031 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:57.857015 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 07 15:46:57.862844 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:57.862822 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5fhk"] Jul 07 15:46:57.912675 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:57.912644 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qpl8\" (UniqueName: \"kubernetes.io/projected/b3c22ee8-1a94-4949-8438-91a59f55ef04-kube-api-access-8qpl8\") pod \"dns-operator-controller-manager-864dd78c4c-b5fhk\" (UID: \"b3c22ee8-1a94-4949-8438-91a59f55ef04\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5fhk" Jul 07 15:46:58.013408 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:58.013372 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qpl8\" (UniqueName: \"kubernetes.io/projected/b3c22ee8-1a94-4949-8438-91a59f55ef04-kube-api-access-8qpl8\") pod \"dns-operator-controller-manager-864dd78c4c-b5fhk\" (UID: \"b3c22ee8-1a94-4949-8438-91a59f55ef04\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5fhk" Jul 07 15:46:58.022487 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:58.022457 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qpl8\" (UniqueName: \"kubernetes.io/projected/b3c22ee8-1a94-4949-8438-91a59f55ef04-kube-api-access-8qpl8\") pod \"dns-operator-controller-manager-864dd78c4c-b5fhk\" (UID: \"b3c22ee8-1a94-4949-8438-91a59f55ef04\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5fhk" Jul 07 15:46:58.166360 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:58.166255 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5fhk" Jul 07 15:46:58.308580 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:58.308554 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5fhk"] Jul 07 15:46:58.310701 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:46:58.310672 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c22ee8_1a94_4949_8438_91a59f55ef04.slice/crio-81b85befe53912e754a3417832a5d0502508e2a20f08329a3e5defd6a3659e5a WatchSource:0}: Error finding container 81b85befe53912e754a3417832a5d0502508e2a20f08329a3e5defd6a3659e5a: Status 404 returned error can't find the container with id 81b85befe53912e754a3417832a5d0502508e2a20f08329a3e5defd6a3659e5a Jul 07 15:46:58.510102 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:46:58.510005 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5fhk" event={"ID":"b3c22ee8-1a94-4949-8438-91a59f55ef04","Type":"ContainerStarted","Data":"81b85befe53912e754a3417832a5d0502508e2a20f08329a3e5defd6a3659e5a"} Jul 07 15:47:01.529652 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:01.529611 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5fhk" event={"ID":"b3c22ee8-1a94-4949-8438-91a59f55ef04","Type":"ContainerStarted","Data":"f01408b1d21b1924dc230fdefb4f0f8738d56c95de15b0d5345c23f22a2e9cd5"} Jul 07 15:47:01.530071 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:01.529743 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5fhk" Jul 07 15:47:01.554522 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:01.554467 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5fhk" podStartSLOduration=2.17503501 podStartE2EDuration="4.554451309s" totalImagesPullingTime="2.379416299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:47:01.553670834 +0000 UTC m=+405.731315285" watchObservedRunningTime="2026-07-07 15:47:01.554451309 +0000 UTC m=+405.732095759" Jul 07 15:47:02.882935 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:02.882896 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx"] Jul 07 15:47:02.891400 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:02.891373 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" Jul 07 15:47:02.894365 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:02.894341 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bh7bf\"" Jul 07 15:47:02.906466 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:02.906302 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx"] Jul 07 15:47:02.958970 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:02.958931 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/710fe1e6-2a32-475c-a556-607addebc8db-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mtkhx\" (UID: \"710fe1e6-2a32-475c-a556-607addebc8db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" Jul 07 15:47:02.958970 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:02.958975 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gndnc\" (UniqueName: \"kubernetes.io/projected/710fe1e6-2a32-475c-a556-607addebc8db-kube-api-access-gndnc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mtkhx\" (UID: \"710fe1e6-2a32-475c-a556-607addebc8db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" Jul 07 15:47:03.060421 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:03.060376 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/710fe1e6-2a32-475c-a556-607addebc8db-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mtkhx\" (UID: \"710fe1e6-2a32-475c-a556-607addebc8db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" Jul 07 15:47:03.060421 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:03.060420 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gndnc\" (UniqueName: \"kubernetes.io/projected/710fe1e6-2a32-475c-a556-607addebc8db-kube-api-access-gndnc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mtkhx\" (UID: \"710fe1e6-2a32-475c-a556-607addebc8db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" Jul 07 15:47:03.060830 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:03.060808 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/710fe1e6-2a32-475c-a556-607addebc8db-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mtkhx\" (UID: \"710fe1e6-2a32-475c-a556-607addebc8db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" Jul 07 15:47:03.070484 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:03.070454 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gndnc\" (UniqueName: \"kubernetes.io/projected/710fe1e6-2a32-475c-a556-607addebc8db-kube-api-access-gndnc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mtkhx\" (UID: \"710fe1e6-2a32-475c-a556-607addebc8db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" Jul 07 15:47:03.210479 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:03.210377 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" Jul 07 15:47:03.395912 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:03.395887 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx"] Jul 07 15:47:03.398352 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:47:03.398319 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod710fe1e6_2a32_475c_a556_607addebc8db.slice/crio-7fe07ccde540953640109383b156fdfa1e574edf9403f98a45a2b94129931336 WatchSource:0}: Error finding container 7fe07ccde540953640109383b156fdfa1e574edf9403f98a45a2b94129931336: Status 404 returned error can't find the container with id 7fe07ccde540953640109383b156fdfa1e574edf9403f98a45a2b94129931336 Jul 07 15:47:03.542274 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:03.542242 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" event={"ID":"710fe1e6-2a32-475c-a556-607addebc8db","Type":"ContainerStarted","Data":"7fe07ccde540953640109383b156fdfa1e574edf9403f98a45a2b94129931336"} Jul 07 15:47:06.723986 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:06.723944 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-mjfpk"] Jul 07 15:47:06.727594 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:06.727575 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mjfpk" Jul 07 15:47:06.729460 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:06.729442 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-4r4w4\"" Jul 07 15:47:06.739278 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:06.739254 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-mjfpk"] Jul 07 15:47:06.792839 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:06.792795 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m7zm\" (UniqueName: \"kubernetes.io/projected/5de713ba-3082-4e16-9015-28a02e8dcb8c-kube-api-access-6m7zm\") pod \"authorino-operator-75cc8d5cb7-mjfpk\" (UID: \"5de713ba-3082-4e16-9015-28a02e8dcb8c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-mjfpk" Jul 07 15:47:06.893667 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:06.893627 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m7zm\" (UniqueName: \"kubernetes.io/projected/5de713ba-3082-4e16-9015-28a02e8dcb8c-kube-api-access-6m7zm\") pod \"authorino-operator-75cc8d5cb7-mjfpk\" (UID: \"5de713ba-3082-4e16-9015-28a02e8dcb8c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-mjfpk" Jul 07 15:47:06.902486 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:06.902453 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m7zm\" (UniqueName: \"kubernetes.io/projected/5de713ba-3082-4e16-9015-28a02e8dcb8c-kube-api-access-6m7zm\") pod \"authorino-operator-75cc8d5cb7-mjfpk\" (UID: \"5de713ba-3082-4e16-9015-28a02e8dcb8c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-mjfpk" Jul 07 15:47:07.039050 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:07.039009 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mjfpk" Jul 07 15:47:07.184622 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:07.184592 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-mjfpk"] Jul 07 15:47:07.187298 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:47:07.187256 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de713ba_3082_4e16_9015_28a02e8dcb8c.slice/crio-bb6a3b2528d203bba9ecd7ebd9b5ac77c2262cb6392e3b150b9ed6fef4f61d38 WatchSource:0}: Error finding container bb6a3b2528d203bba9ecd7ebd9b5ac77c2262cb6392e3b150b9ed6fef4f61d38: Status 404 returned error can't find the container with id bb6a3b2528d203bba9ecd7ebd9b5ac77c2262cb6392e3b150b9ed6fef4f61d38 Jul 07 15:47:07.565675 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:07.565636 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mjfpk" event={"ID":"5de713ba-3082-4e16-9015-28a02e8dcb8c","Type":"ContainerStarted","Data":"bb6a3b2528d203bba9ecd7ebd9b5ac77c2262cb6392e3b150b9ed6fef4f61d38"} Jul 07 15:47:08.513060 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:08.513023 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z" Jul 07 15:47:11.593323 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:11.593274 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mjfpk" event={"ID":"5de713ba-3082-4e16-9015-28a02e8dcb8c","Type":"ContainerStarted","Data":"04586011f523b8090655fa312aa2b9e5a0e73025746bc3d5f814d3f85ff1dd00"} Jul 07 15:47:11.593826 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:11.593350 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mjfpk" Jul 07 15:47:11.595015 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:11.594980 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" event={"ID":"710fe1e6-2a32-475c-a556-607addebc8db","Type":"ContainerStarted","Data":"b2024269f1e2434a76494a80aa5fc10fbd9b3375871c244206a3d5d8c780bb7a"} Jul 07 15:47:11.595175 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:11.595079 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" Jul 07 15:47:11.615230 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:11.615146 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mjfpk" podStartSLOduration=2.161913644 podStartE2EDuration="5.615096509s" totalImagesPullingTime="3.453182865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:47:11.612289503 +0000 UTC m=+415.789933954" watchObservedRunningTime="2026-07-07 15:47:11.615096509 +0000 UTC m=+415.792740959" Jul 07 15:47:11.635093 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:11.635036 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" podStartSLOduration=2.434384644 podStartE2EDuration="9.635019998s" totalImagesPullingTime="7.200635354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:47:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:47:11.634213473 +0000 UTC m=+415.811857947" watchObservedRunningTime="2026-07-07 15:47:11.635019998 +0000 UTC m=+415.812664447" Jul 07 15:47:12.536916 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:12.536879 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b5fhk" Jul 07 15:47:13.671130 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:13.671079 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qxpkg"] Jul 07 15:47:13.676340 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:13.676317 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxpkg" Jul 07 15:47:13.679900 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:13.679875 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 07 15:47:13.680023 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:13.680008 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 07 15:47:13.680078 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:13.680042 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gnpsb\"" Jul 07 15:47:13.688679 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:13.688652 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qxpkg"] Jul 07 15:47:13.758789 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:13.758752 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4184c3b8-50a2-4d79-8afa-0eefe6132943-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qxpkg\" (UID: \"4184c3b8-50a2-4d79-8afa-0eefe6132943\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxpkg" Jul 07 15:47:13.759000 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:13.758812 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvm62\" (UniqueName: \"kubernetes.io/projected/4184c3b8-50a2-4d79-8afa-0eefe6132943-kube-api-access-vvm62\") pod \"kuadrant-console-plugin-66b5774985-qxpkg\" (UID: \"4184c3b8-50a2-4d79-8afa-0eefe6132943\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxpkg" Jul 07 15:47:13.759000 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:13.758909 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4184c3b8-50a2-4d79-8afa-0eefe6132943-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qxpkg\" (UID: \"4184c3b8-50a2-4d79-8afa-0eefe6132943\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxpkg" Jul 07 15:47:13.859813 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:13.859771 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4184c3b8-50a2-4d79-8afa-0eefe6132943-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qxpkg\" (UID: \"4184c3b8-50a2-4d79-8afa-0eefe6132943\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxpkg" Jul 07 15:47:13.860059 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:13.859852 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvm62\" (UniqueName: \"kubernetes.io/projected/4184c3b8-50a2-4d79-8afa-0eefe6132943-kube-api-access-vvm62\") pod \"kuadrant-console-plugin-66b5774985-qxpkg\" (UID: \"4184c3b8-50a2-4d79-8afa-0eefe6132943\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxpkg" Jul 07 15:47:13.860059 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:13.859904 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4184c3b8-50a2-4d79-8afa-0eefe6132943-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qxpkg\" (UID: \"4184c3b8-50a2-4d79-8afa-0eefe6132943\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxpkg" Jul 07 15:47:13.860059 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:13.860020 2557 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 07 15:47:13.860349 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:13.860089 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4184c3b8-50a2-4d79-8afa-0eefe6132943-plugin-serving-cert podName:4184c3b8-50a2-4d79-8afa-0eefe6132943 nodeName:}" failed. No retries permitted until 2026-07-07 15:47:14.360068025 +0000 UTC m=+418.537712453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/4184c3b8-50a2-4d79-8afa-0eefe6132943-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-qxpkg" (UID: "4184c3b8-50a2-4d79-8afa-0eefe6132943") : secret "plugin-serving-cert" not found Jul 07 15:47:13.860675 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:13.860644 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4184c3b8-50a2-4d79-8afa-0eefe6132943-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qxpkg\" (UID: \"4184c3b8-50a2-4d79-8afa-0eefe6132943\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxpkg" Jul 07 15:47:13.870749 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:13.870721 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvm62\" (UniqueName: \"kubernetes.io/projected/4184c3b8-50a2-4d79-8afa-0eefe6132943-kube-api-access-vvm62\") pod \"kuadrant-console-plugin-66b5774985-qxpkg\" (UID: \"4184c3b8-50a2-4d79-8afa-0eefe6132943\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxpkg" Jul 07 15:47:14.364916 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:14.364880 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4184c3b8-50a2-4d79-8afa-0eefe6132943-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qxpkg\" (UID: \"4184c3b8-50a2-4d79-8afa-0eefe6132943\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxpkg" Jul 07 15:47:14.367469 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:14.367445 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4184c3b8-50a2-4d79-8afa-0eefe6132943-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qxpkg\" (UID: \"4184c3b8-50a2-4d79-8afa-0eefe6132943\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxpkg" Jul 07 15:47:14.586248 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:14.586213 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxpkg" Jul 07 15:47:14.736373 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:14.736347 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qxpkg"] Jul 07 15:47:14.738485 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:47:14.738459 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4184c3b8_50a2_4d79_8afa_0eefe6132943.slice/crio-c2df805abf72843d183e31cd0bb7c00a02f3b3a993e411cfa2a850e3c19c2270 WatchSource:0}: Error finding container c2df805abf72843d183e31cd0bb7c00a02f3b3a993e411cfa2a850e3c19c2270: Status 404 returned error can't find the container with id c2df805abf72843d183e31cd0bb7c00a02f3b3a993e411cfa2a850e3c19c2270 Jul 07 15:47:15.628142 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:15.628088 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxpkg" event={"ID":"4184c3b8-50a2-4d79-8afa-0eefe6132943","Type":"ContainerStarted","Data":"c2df805abf72843d183e31cd0bb7c00a02f3b3a993e411cfa2a850e3c19c2270"} Jul 07 15:47:22.603605 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:22.603568 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" Jul 07 15:47:22.603990 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:22.603747 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mjfpk" Jul 07 15:47:23.894241 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:23.894200 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-swk5m"] Jul 07 15:47:23.898495 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:23.898474 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-swk5m" Jul 07 15:47:23.914318 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:23.914290 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-swk5m"] Jul 07 15:47:24.053159 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.053123 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78820751-acd3-4a2e-b600-f19e89435a8a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-swk5m\" (UID: \"78820751-acd3-4a2e-b600-f19e89435a8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-swk5m" Jul 07 15:47:24.053355 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.053193 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc72g\" (UniqueName: \"kubernetes.io/projected/78820751-acd3-4a2e-b600-f19e89435a8a-kube-api-access-lc72g\") pod \"kuadrant-operator-controller-manager-55467d9f7-swk5m\" (UID: \"78820751-acd3-4a2e-b600-f19e89435a8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-swk5m" Jul 07 15:47:24.154542 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.154443 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78820751-acd3-4a2e-b600-f19e89435a8a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-swk5m\" (UID: \"78820751-acd3-4a2e-b600-f19e89435a8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-swk5m" Jul 07 15:47:24.154542 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.154502 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc72g\" (UniqueName: \"kubernetes.io/projected/78820751-acd3-4a2e-b600-f19e89435a8a-kube-api-access-lc72g\") pod \"kuadrant-operator-controller-manager-55467d9f7-swk5m\" (UID: \"78820751-acd3-4a2e-b600-f19e89435a8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-swk5m" Jul 07 15:47:24.154921 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.154897 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78820751-acd3-4a2e-b600-f19e89435a8a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-swk5m\" (UID: \"78820751-acd3-4a2e-b600-f19e89435a8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-swk5m" Jul 07 15:47:24.165782 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.165753 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc72g\" (UniqueName: \"kubernetes.io/projected/78820751-acd3-4a2e-b600-f19e89435a8a-kube-api-access-lc72g\") pod \"kuadrant-operator-controller-manager-55467d9f7-swk5m\" (UID: \"78820751-acd3-4a2e-b600-f19e89435a8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-swk5m" Jul 07 15:47:24.211614 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.211563 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-swk5m" Jul 07 15:47:24.328214 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.328181 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-swk5m"] Jul 07 15:47:24.339510 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.339477 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-swk5m"] Jul 07 15:47:24.343871 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:47:24.343825 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78820751_acd3_4a2e_b600_f19e89435a8a.slice/crio-6822ce86a781925f8cbe4e988bb28c6905ac06321892a8febdb5cc99a3a8011d WatchSource:0}: Error finding container 6822ce86a781925f8cbe4e988bb28c6905ac06321892a8febdb5cc99a3a8011d: Status 404 returned error can't find the container with id 6822ce86a781925f8cbe4e988bb28c6905ac06321892a8febdb5cc99a3a8011d Jul 07 15:47:24.359541 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.359435 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm"] Jul 07 15:47:24.364217 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.364148 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm" Jul 07 15:47:24.379901 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.379825 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm"] Jul 07 15:47:24.383678 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.383652 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx"] Jul 07 15:47:24.383844 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.383825 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" podUID="710fe1e6-2a32-475c-a556-607addebc8db" containerName="manager" containerID="cri-o://b2024269f1e2434a76494a80aa5fc10fbd9b3375871c244206a3d5d8c780bb7a" gracePeriod=2 Jul 07 15:47:24.394793 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.394707 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx"] Jul 07 15:47:24.399655 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:24.399309 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mtkhx\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" podUID="710fe1e6-2a32-475c-a556-607addebc8db" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" Jul 07 15:47:24.406020 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.405953 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z"] Jul 07 15:47:24.406194 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.406170 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z" podUID="b1bbbb98-c897-409a-8667-14c39593c63e" containerName="manager" containerID="cri-o://18e7d4396231c4df11f3c77ea19e838c230847abb261cfcc68f555dd15534a62" gracePeriod=2 Jul 07 15:47:24.411883 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:24.410244 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mtkhx\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" podUID="710fe1e6-2a32-475c-a556-607addebc8db" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" Jul 07 15:47:24.424260 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.424231 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z"] Jul 07 15:47:24.448375 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:24.448318 2557 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"b1bbbb98-c897-409a-8667-14c39593c63e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-q7l7z\": pods \"limitador-operator-controller-manager-859d7b78bd-q7l7z\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z" Jul 07 15:47:24.456810 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.456494 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jcw8t"] Jul 07 15:47:24.457625 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.457545 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1bbbb98-c897-409a-8667-14c39593c63e" containerName="manager" Jul 07 15:47:24.457625 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.457571 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1bbbb98-c897-409a-8667-14c39593c63e" containerName="manager" Jul 07 15:47:24.457625 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.457611 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="710fe1e6-2a32-475c-a556-607addebc8db" containerName="manager" Jul 07 15:47:24.457625 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.457621 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="710fe1e6-2a32-475c-a556-607addebc8db" containerName="manager" Jul 07 15:47:24.458414 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.457753 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="710fe1e6-2a32-475c-a556-607addebc8db" containerName="manager" Jul 07 15:47:24.458414 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.457767 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1bbbb98-c897-409a-8667-14c39593c63e" containerName="manager" Jul 07 15:47:24.461751 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.461728 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jcw8t" Jul 07 15:47:24.474710 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.474681 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jcw8t"] Jul 07 15:47:24.492995 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:24.492838 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mtkhx\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" podUID="710fe1e6-2a32-475c-a556-607addebc8db" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" Jul 07 15:47:24.495537 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:24.495508 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q7l7z\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" podUID="b1bbbb98-c897-409a-8667-14c39593c63e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z" Jul 07 15:47:24.559751 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.559722 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17fddd64-4b62-4d34-b90d-6088bc1ae3c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-q8wxm\" (UID: \"17fddd64-4b62-4d34-b90d-6088bc1ae3c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm" Jul 07 15:47:24.559896 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.559780 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb94v\" (UniqueName: \"kubernetes.io/projected/6d816f57-931a-4919-91c6-fec98b115825-kube-api-access-zb94v\") pod \"limitador-operator-controller-manager-859d7b78bd-jcw8t\" (UID: \"6d816f57-931a-4919-91c6-fec98b115825\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jcw8t" Jul 07 15:47:24.559896 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.559825 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsllm\" (UniqueName: \"kubernetes.io/projected/17fddd64-4b62-4d34-b90d-6088bc1ae3c6-kube-api-access-wsllm\") pod \"kuadrant-operator-controller-manager-55467d9f7-q8wxm\" (UID: \"17fddd64-4b62-4d34-b90d-6088bc1ae3c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm" Jul 07 15:47:24.653537 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.653510 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" Jul 07 15:47:24.655506 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:24.655469 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mtkhx\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" podUID="710fe1e6-2a32-475c-a556-607addebc8db" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" Jul 07 15:47:24.657218 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:24.657189 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q7l7z\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" podUID="b1bbbb98-c897-409a-8667-14c39593c63e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z" Jul 07 15:47:24.660557 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.660531 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17fddd64-4b62-4d34-b90d-6088bc1ae3c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-q8wxm\" (UID: \"17fddd64-4b62-4d34-b90d-6088bc1ae3c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm" Jul 07 15:47:24.660731 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.660581 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb94v\" (UniqueName: \"kubernetes.io/projected/6d816f57-931a-4919-91c6-fec98b115825-kube-api-access-zb94v\") pod \"limitador-operator-controller-manager-859d7b78bd-jcw8t\" (UID: \"6d816f57-931a-4919-91c6-fec98b115825\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jcw8t" Jul 07 15:47:24.660731 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.660629 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsllm\" (UniqueName: \"kubernetes.io/projected/17fddd64-4b62-4d34-b90d-6088bc1ae3c6-kube-api-access-wsllm\") pod \"kuadrant-operator-controller-manager-55467d9f7-q8wxm\" (UID: \"17fddd64-4b62-4d34-b90d-6088bc1ae3c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm" Jul 07 15:47:24.660956 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.660934 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17fddd64-4b62-4d34-b90d-6088bc1ae3c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-q8wxm\" (UID: \"17fddd64-4b62-4d34-b90d-6088bc1ae3c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm" Jul 07 15:47:24.663931 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.663912 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z" Jul 07 15:47:24.668340 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:24.668313 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mtkhx\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" podUID="710fe1e6-2a32-475c-a556-607addebc8db" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" Jul 07 15:47:24.669797 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:24.669774 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q7l7z\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" podUID="b1bbbb98-c897-409a-8667-14c39593c63e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z" Jul 07 15:47:24.679528 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.679502 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsllm\" (UniqueName: \"kubernetes.io/projected/17fddd64-4b62-4d34-b90d-6088bc1ae3c6-kube-api-access-wsllm\") pod \"kuadrant-operator-controller-manager-55467d9f7-q8wxm\" (UID: \"17fddd64-4b62-4d34-b90d-6088bc1ae3c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm" Jul 07 15:47:24.683392 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.683366 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb94v\" (UniqueName: \"kubernetes.io/projected/6d816f57-931a-4919-91c6-fec98b115825-kube-api-access-zb94v\") pod \"limitador-operator-controller-manager-859d7b78bd-jcw8t\" (UID: \"6d816f57-931a-4919-91c6-fec98b115825\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jcw8t" Jul 07 15:47:24.691723 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.691687 2557 generic.go:356] "Generic (PLEG): container finished" podID="b1bbbb98-c897-409a-8667-14c39593c63e" containerID="18e7d4396231c4df11f3c77ea19e838c230847abb261cfcc68f555dd15534a62" exitCode=0 Jul 07 15:47:24.691887 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.691752 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z" Jul 07 15:47:24.691887 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.691780 2557 scope.go:117] "RemoveContainer" containerID="18e7d4396231c4df11f3c77ea19e838c230847abb261cfcc68f555dd15534a62" Jul 07 15:47:24.693856 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.693813 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-swk5m" podUID="78820751-acd3-4a2e-b600-f19e89435a8a" containerName="manager" containerID="cri-o://23858468af47d69b4415d6a29d0c8b376e37f4a2dd994ce95ac40f7fd7764f70" gracePeriod=2 Jul 07 15:47:24.694216 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:24.694178 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mtkhx\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" podUID="710fe1e6-2a32-475c-a556-607addebc8db" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" Jul 07 15:47:24.695369 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.695348 2557 generic.go:356] "Generic (PLEG): container finished" podID="710fe1e6-2a32-475c-a556-607addebc8db" containerID="b2024269f1e2434a76494a80aa5fc10fbd9b3375871c244206a3d5d8c780bb7a" exitCode=0 Jul 07 15:47:24.695483 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.695392 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" Jul 07 15:47:24.701075 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.701045 2557 scope.go:117] "RemoveContainer" containerID="18e7d4396231c4df11f3c77ea19e838c230847abb261cfcc68f555dd15534a62" Jul 07 15:47:24.701363 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:24.701344 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e7d4396231c4df11f3c77ea19e838c230847abb261cfcc68f555dd15534a62\": container with ID starting with 18e7d4396231c4df11f3c77ea19e838c230847abb261cfcc68f555dd15534a62 not found: ID does not exist" containerID="18e7d4396231c4df11f3c77ea19e838c230847abb261cfcc68f555dd15534a62" Jul 07 15:47:24.701426 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.701372 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e7d4396231c4df11f3c77ea19e838c230847abb261cfcc68f555dd15534a62"} err="failed to get container status \"18e7d4396231c4df11f3c77ea19e838c230847abb261cfcc68f555dd15534a62\": rpc error: code = NotFound desc = could not find container \"18e7d4396231c4df11f3c77ea19e838c230847abb261cfcc68f555dd15534a62\": container with ID starting with 18e7d4396231c4df11f3c77ea19e838c230847abb261cfcc68f555dd15534a62 not found: ID does not exist" Jul 07 15:47:24.701426 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.701390 2557 scope.go:117] "RemoveContainer" containerID="b2024269f1e2434a76494a80aa5fc10fbd9b3375871c244206a3d5d8c780bb7a" Jul 07 15:47:24.705054 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:24.705029 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q7l7z\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" podUID="b1bbbb98-c897-409a-8667-14c39593c63e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z" Jul 07 15:47:24.706699 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:24.706677 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q7l7z\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" podUID="b1bbbb98-c897-409a-8667-14c39593c63e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z" Jul 07 15:47:24.708625 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:24.708603 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mtkhx\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" podUID="710fe1e6-2a32-475c-a556-607addebc8db" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" Jul 07 15:47:24.709691 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.709668 2557 scope.go:117] "RemoveContainer" containerID="b2024269f1e2434a76494a80aa5fc10fbd9b3375871c244206a3d5d8c780bb7a" Jul 07 15:47:24.709961 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:24.709938 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2024269f1e2434a76494a80aa5fc10fbd9b3375871c244206a3d5d8c780bb7a\": container with ID starting with b2024269f1e2434a76494a80aa5fc10fbd9b3375871c244206a3d5d8c780bb7a not found: ID does not exist" containerID="b2024269f1e2434a76494a80aa5fc10fbd9b3375871c244206a3d5d8c780bb7a" Jul 07 15:47:24.710001 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.709972 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2024269f1e2434a76494a80aa5fc10fbd9b3375871c244206a3d5d8c780bb7a"} err="failed to get container status \"b2024269f1e2434a76494a80aa5fc10fbd9b3375871c244206a3d5d8c780bb7a\": rpc error: code = NotFound desc = could not find container \"b2024269f1e2434a76494a80aa5fc10fbd9b3375871c244206a3d5d8c780bb7a\": container with ID starting with b2024269f1e2434a76494a80aa5fc10fbd9b3375871c244206a3d5d8c780bb7a not found: ID does not exist" Jul 07 15:47:24.761693 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.761657 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gndnc\" (UniqueName: \"kubernetes.io/projected/710fe1e6-2a32-475c-a556-607addebc8db-kube-api-access-gndnc\") pod \"710fe1e6-2a32-475c-a556-607addebc8db\" (UID: \"710fe1e6-2a32-475c-a556-607addebc8db\") " Jul 07 15:47:24.761897 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.761813 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/710fe1e6-2a32-475c-a556-607addebc8db-extensions-socket-volume\") pod \"710fe1e6-2a32-475c-a556-607addebc8db\" (UID: \"710fe1e6-2a32-475c-a556-607addebc8db\") " Jul 07 15:47:24.762318 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.762294 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710fe1e6-2a32-475c-a556-607addebc8db-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "710fe1e6-2a32-475c-a556-607addebc8db" (UID: "710fe1e6-2a32-475c-a556-607addebc8db"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:47:24.763901 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.763878 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/710fe1e6-2a32-475c-a556-607addebc8db-kube-api-access-gndnc" (OuterVolumeSpecName: "kube-api-access-gndnc") pod "710fe1e6-2a32-475c-a556-607addebc8db" (UID: "710fe1e6-2a32-475c-a556-607addebc8db"). InnerVolumeSpecName "kube-api-access-gndnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:47:24.833775 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.833736 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm" Jul 07 15:47:24.859662 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.859620 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jcw8t" Jul 07 15:47:24.862987 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.862938 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xx9mt\" (UniqueName: \"kubernetes.io/projected/b1bbbb98-c897-409a-8667-14c39593c63e-kube-api-access-xx9mt\") pod \"b1bbbb98-c897-409a-8667-14c39593c63e\" (UID: \"b1bbbb98-c897-409a-8667-14c39593c63e\") " Jul 07 15:47:24.867217 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.864103 2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/710fe1e6-2a32-475c-a556-607addebc8db-extensions-socket-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:47:24.867217 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.864176 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gndnc\" (UniqueName: \"kubernetes.io/projected/710fe1e6-2a32-475c-a556-607addebc8db-kube-api-access-gndnc\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:47:24.871053 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.871016 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1bbbb98-c897-409a-8667-14c39593c63e-kube-api-access-xx9mt" (OuterVolumeSpecName: "kube-api-access-xx9mt") pod "b1bbbb98-c897-409a-8667-14c39593c63e" (UID: "b1bbbb98-c897-409a-8667-14c39593c63e"). InnerVolumeSpecName "kube-api-access-xx9mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:47:24.965215 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:24.965096 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx9mt\" (UniqueName: \"kubernetes.io/projected/b1bbbb98-c897-409a-8667-14c39593c63e-kube-api-access-xx9mt\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:47:25.006418 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:25.006374 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mtkhx\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" podUID="710fe1e6-2a32-475c-a556-607addebc8db" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" Jul 07 15:47:25.008949 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:25.008910 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q7l7z\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" podUID="b1bbbb98-c897-409a-8667-14c39593c63e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z" Jul 07 15:47:25.010517 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:25.010484 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mtkhx\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" podUID="710fe1e6-2a32-475c-a556-607addebc8db" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" Jul 07 15:47:25.011980 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:25.011945 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q7l7z\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" podUID="b1bbbb98-c897-409a-8667-14c39593c63e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z" Jul 07 15:47:26.350161 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:26.350105 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q7l7z\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" podUID="b1bbbb98-c897-409a-8667-14c39593c63e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q7l7z" Jul 07 15:47:26.352046 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:26.352005 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mtkhx\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" podUID="710fe1e6-2a32-475c-a556-607addebc8db" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mtkhx" Jul 07 15:47:26.352230 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:26.352075 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="710fe1e6-2a32-475c-a556-607addebc8db" path="/var/lib/kubelet/pods/710fe1e6-2a32-475c-a556-607addebc8db/volumes" Jul 07 15:47:26.352538 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:26.352519 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1bbbb98-c897-409a-8667-14c39593c63e" path="/var/lib/kubelet/pods/b1bbbb98-c897-409a-8667-14c39593c63e/volumes" Jul 07 15:47:39.444265 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.444197 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-swk5m" Jul 07 15:47:39.446596 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:39.446558 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-swk5m\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" podUID="78820751-acd3-4a2e-b600-f19e89435a8a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-swk5m" Jul 07 15:47:39.456971 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.456909 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jcw8t"] Jul 07 15:47:39.464076 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:47:39.464050 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d816f57_931a_4919_91c6_fec98b115825.slice/crio-fe846596996f02255ca7952117cfce1046d399028d1865df606d58b75c5d15f3 WatchSource:0}: Error finding container fe846596996f02255ca7952117cfce1046d399028d1865df606d58b75c5d15f3: Status 404 returned error can't find the container with id fe846596996f02255ca7952117cfce1046d399028d1865df606d58b75c5d15f3 Jul 07 15:47:39.481364 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.481331 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm"] Jul 07 15:47:39.483145 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:47:39.483098 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fddd64_4b62_4d34_b90d_6088bc1ae3c6.slice/crio-33251398c1d36963d2bd2ea0817c8885e623797b38ff8329a581db5a01494482 WatchSource:0}: Error finding container 33251398c1d36963d2bd2ea0817c8885e623797b38ff8329a581db5a01494482: Status 404 returned error can't find the container with id 33251398c1d36963d2bd2ea0817c8885e623797b38ff8329a581db5a01494482 Jul 07 15:47:39.503354 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.503327 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78820751-acd3-4a2e-b600-f19e89435a8a-extensions-socket-volume\") pod \"78820751-acd3-4a2e-b600-f19e89435a8a\" (UID: \"78820751-acd3-4a2e-b600-f19e89435a8a\") " Jul 07 15:47:39.503476 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.503391 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc72g\" (UniqueName: \"kubernetes.io/projected/78820751-acd3-4a2e-b600-f19e89435a8a-kube-api-access-lc72g\") pod \"78820751-acd3-4a2e-b600-f19e89435a8a\" (UID: \"78820751-acd3-4a2e-b600-f19e89435a8a\") " Jul 07 15:47:39.503640 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.503611 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78820751-acd3-4a2e-b600-f19e89435a8a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "78820751-acd3-4a2e-b600-f19e89435a8a" (UID: "78820751-acd3-4a2e-b600-f19e89435a8a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:47:39.505454 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.505432 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78820751-acd3-4a2e-b600-f19e89435a8a-kube-api-access-lc72g" (OuterVolumeSpecName: "kube-api-access-lc72g") pod "78820751-acd3-4a2e-b600-f19e89435a8a" (UID: "78820751-acd3-4a2e-b600-f19e89435a8a"). InnerVolumeSpecName "kube-api-access-lc72g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:47:39.604461 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.604432 2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78820751-acd3-4a2e-b600-f19e89435a8a-extensions-socket-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:47:39.604461 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.604459 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc72g\" (UniqueName: \"kubernetes.io/projected/78820751-acd3-4a2e-b600-f19e89435a8a-kube-api-access-lc72g\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:47:39.802500 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.802460 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxpkg" event={"ID":"4184c3b8-50a2-4d79-8afa-0eefe6132943","Type":"ContainerStarted","Data":"c75210c2cf2c3831f33956cfc4745d4e956803f1787a6e292b51e6c851ffef73"} Jul 07 15:47:39.804083 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.804056 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm" event={"ID":"17fddd64-4b62-4d34-b90d-6088bc1ae3c6","Type":"ContainerStarted","Data":"ecc74922f8db98e988df19f3ad957c0f7cf269ab23279660c941ac9873fd2ee9"} Jul 07 15:47:39.804238 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.804136 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm" event={"ID":"17fddd64-4b62-4d34-b90d-6088bc1ae3c6","Type":"ContainerStarted","Data":"33251398c1d36963d2bd2ea0817c8885e623797b38ff8329a581db5a01494482"} Jul 07 15:47:39.804238 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.804182 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm" Jul 07 15:47:39.804356 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:39.804249 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-swk5m\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" podUID="78820751-acd3-4a2e-b600-f19e89435a8a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-swk5m" Jul 07 15:47:39.805450 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.805430 2557 generic.go:356] "Generic (PLEG): container finished" podID="78820751-acd3-4a2e-b600-f19e89435a8a" containerID="23858468af47d69b4415d6a29d0c8b376e37f4a2dd994ce95ac40f7fd7764f70" exitCode=2 Jul 07 15:47:39.805544 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.805467 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-swk5m" Jul 07 15:47:39.805544 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.805532 2557 scope.go:117] "RemoveContainer" containerID="23858468af47d69b4415d6a29d0c8b376e37f4a2dd994ce95ac40f7fd7764f70" Jul 07 15:47:39.807352 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.807313 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jcw8t" event={"ID":"6d816f57-931a-4919-91c6-fec98b115825","Type":"ContainerStarted","Data":"373d4ef708661f3656695c3f6c97e71d5b8259aeba36c0de0e25fcc51295ac7b"} Jul 07 15:47:39.807437 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.807363 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jcw8t" event={"ID":"6d816f57-931a-4919-91c6-fec98b115825","Type":"ContainerStarted","Data":"fe846596996f02255ca7952117cfce1046d399028d1865df606d58b75c5d15f3"} Jul 07 15:47:39.807437 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.807397 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jcw8t" Jul 07 15:47:39.815064 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.815047 2557 scope.go:117] "RemoveContainer" containerID="23858468af47d69b4415d6a29d0c8b376e37f4a2dd994ce95ac40f7fd7764f70" Jul 07 15:47:39.815361 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:39.815344 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23858468af47d69b4415d6a29d0c8b376e37f4a2dd994ce95ac40f7fd7764f70\": container with ID starting with 23858468af47d69b4415d6a29d0c8b376e37f4a2dd994ce95ac40f7fd7764f70 not found: ID does not exist" containerID="23858468af47d69b4415d6a29d0c8b376e37f4a2dd994ce95ac40f7fd7764f70" Jul 07 15:47:39.815437 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.815375 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23858468af47d69b4415d6a29d0c8b376e37f4a2dd994ce95ac40f7fd7764f70"} err="failed to get container status \"23858468af47d69b4415d6a29d0c8b376e37f4a2dd994ce95ac40f7fd7764f70\": rpc error: code = NotFound desc = could not find container \"23858468af47d69b4415d6a29d0c8b376e37f4a2dd994ce95ac40f7fd7764f70\": container with ID starting with 23858468af47d69b4415d6a29d0c8b376e37f4a2dd994ce95ac40f7fd7764f70 not found: ID does not exist" Jul 07 15:47:39.822388 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.822345 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxpkg" podStartSLOduration=2.191198435 podStartE2EDuration="26.822332455s" totalImagesPullingTime="24.63113402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:47:39.820515234 +0000 UTC m=+443.998159686" watchObservedRunningTime="2026-07-07 15:47:39.822332455 +0000 UTC m=+443.999976905" Jul 07 15:47:39.838850 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.838469 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jcw8t" podStartSLOduration=15.838451288 podStartE2EDuration="15.838451288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:47:39.837945386 +0000 UTC m=+444.015589896" watchObservedRunningTime="2026-07-07 15:47:39.838451288 +0000 UTC m=+444.016095738" Jul 07 15:47:39.859852 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:47:39.859794 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-swk5m\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" podUID="78820751-acd3-4a2e-b600-f19e89435a8a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-swk5m" Jul 07 15:47:39.860704 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:39.860663 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm" podStartSLOduration=15.860648489 podStartE2EDuration="15.860648489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:47:39.858189149 +0000 UTC m=+444.035833626" watchObservedRunningTime="2026-07-07 15:47:39.860648489 +0000 UTC m=+444.038292938" Jul 07 15:47:40.354053 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:40.354017 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78820751-acd3-4a2e-b600-f19e89435a8a" path="/var/lib/kubelet/pods/78820751-acd3-4a2e-b600-f19e89435a8a/volumes" Jul 07 15:47:50.815834 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:50.815801 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jcw8t" Jul 07 15:47:50.816350 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:50.815919 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm" Jul 07 15:47:51.202566 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:51.202527 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm"] Jul 07 15:47:51.202920 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:51.202908 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78820751-acd3-4a2e-b600-f19e89435a8a" containerName="manager" Jul 07 15:47:51.202972 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:51.202921 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="78820751-acd3-4a2e-b600-f19e89435a8a" containerName="manager" Jul 07 15:47:51.203014 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:51.202995 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="78820751-acd3-4a2e-b600-f19e89435a8a" containerName="manager" Jul 07 15:47:51.215065 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:51.215034 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm" Jul 07 15:47:51.224660 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:51.224627 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm"] Jul 07 15:47:51.309198 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:51.309155 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72pjn\" (UniqueName: \"kubernetes.io/projected/c45aca8d-b099-4efa-a2bb-1f5138cc072b-kube-api-access-72pjn\") pod \"kuadrant-operator-controller-manager-86878c769f-5q2qm\" (UID: \"c45aca8d-b099-4efa-a2bb-1f5138cc072b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm" Jul 07 15:47:51.309406 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:51.309239 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c45aca8d-b099-4efa-a2bb-1f5138cc072b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5q2qm\" (UID: \"c45aca8d-b099-4efa-a2bb-1f5138cc072b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm" Jul 07 15:47:51.410620 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:51.410520 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c45aca8d-b099-4efa-a2bb-1f5138cc072b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5q2qm\" (UID: \"c45aca8d-b099-4efa-a2bb-1f5138cc072b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm" Jul 07 15:47:51.410802 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:51.410631 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72pjn\" (UniqueName: \"kubernetes.io/projected/c45aca8d-b099-4efa-a2bb-1f5138cc072b-kube-api-access-72pjn\") pod \"kuadrant-operator-controller-manager-86878c769f-5q2qm\" (UID: \"c45aca8d-b099-4efa-a2bb-1f5138cc072b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm" Jul 07 15:47:51.410925 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:51.410904 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c45aca8d-b099-4efa-a2bb-1f5138cc072b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5q2qm\" (UID: \"c45aca8d-b099-4efa-a2bb-1f5138cc072b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm" Jul 07 15:47:51.418619 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:51.418589 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72pjn\" (UniqueName: \"kubernetes.io/projected/c45aca8d-b099-4efa-a2bb-1f5138cc072b-kube-api-access-72pjn\") pod \"kuadrant-operator-controller-manager-86878c769f-5q2qm\" (UID: \"c45aca8d-b099-4efa-a2bb-1f5138cc072b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm" Jul 07 15:47:51.527702 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:51.527647 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm" Jul 07 15:47:51.691020 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:51.690992 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm"] Jul 07 15:47:51.691646 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:47:51.691610 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45aca8d_b099_4efa_a2bb_1f5138cc072b.slice/crio-8eab9efcb0b8e5df314747c7cc279f34e73aaae085e673395ae0193472f7f2f1 WatchSource:0}: Error finding container 8eab9efcb0b8e5df314747c7cc279f34e73aaae085e673395ae0193472f7f2f1: Status 404 returned error can't find the container with id 8eab9efcb0b8e5df314747c7cc279f34e73aaae085e673395ae0193472f7f2f1 Jul 07 15:47:51.878726 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:51.878686 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm" event={"ID":"c45aca8d-b099-4efa-a2bb-1f5138cc072b","Type":"ContainerStarted","Data":"62917a4d9c8f935b9bb9c1d4381fb46a9e1bd8b9449dbb0c793678e291f3d18b"} Jul 07 15:47:51.879201 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:51.878740 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm" event={"ID":"c45aca8d-b099-4efa-a2bb-1f5138cc072b","Type":"ContainerStarted","Data":"8eab9efcb0b8e5df314747c7cc279f34e73aaae085e673395ae0193472f7f2f1"} Jul 07 15:47:51.879201 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:51.878775 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm" Jul 07 15:47:51.912739 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:47:51.912680 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm" podStartSLOduration=0.912660189 podStartE2EDuration="912.660189ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:47:51.910978214 +0000 UTC m=+456.088622663" watchObservedRunningTime="2026-07-07 15:47:51.912660189 +0000 UTC m=+456.090304639" Jul 07 15:48:02.887990 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:02.887956 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm" Jul 07 15:48:02.937701 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:02.937656 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm"] Jul 07 15:48:02.938106 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:02.938038 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm" podUID="17fddd64-4b62-4d34-b90d-6088bc1ae3c6" containerName="manager" containerID="cri-o://ecc74922f8db98e988df19f3ad957c0f7cf269ab23279660c941ac9873fd2ee9" gracePeriod=10 Jul 07 15:48:03.823287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:03.823250 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm" Jul 07 15:48:03.922356 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:03.922264 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsllm\" (UniqueName: \"kubernetes.io/projected/17fddd64-4b62-4d34-b90d-6088bc1ae3c6-kube-api-access-wsllm\") pod \"17fddd64-4b62-4d34-b90d-6088bc1ae3c6\" (UID: \"17fddd64-4b62-4d34-b90d-6088bc1ae3c6\") " Jul 07 15:48:03.922826 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:03.922364 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17fddd64-4b62-4d34-b90d-6088bc1ae3c6-extensions-socket-volume\") pod \"17fddd64-4b62-4d34-b90d-6088bc1ae3c6\" (UID: \"17fddd64-4b62-4d34-b90d-6088bc1ae3c6\") " Jul 07 15:48:03.922826 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:03.922739 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fddd64-4b62-4d34-b90d-6088bc1ae3c6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "17fddd64-4b62-4d34-b90d-6088bc1ae3c6" (UID: "17fddd64-4b62-4d34-b90d-6088bc1ae3c6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:48:03.924608 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:03.924561 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fddd64-4b62-4d34-b90d-6088bc1ae3c6-kube-api-access-wsllm" (OuterVolumeSpecName: "kube-api-access-wsllm") pod "17fddd64-4b62-4d34-b90d-6088bc1ae3c6" (UID: "17fddd64-4b62-4d34-b90d-6088bc1ae3c6"). InnerVolumeSpecName "kube-api-access-wsllm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:48:03.951255 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:03.951214 2557 generic.go:356] "Generic (PLEG): container finished" podID="17fddd64-4b62-4d34-b90d-6088bc1ae3c6" containerID="ecc74922f8db98e988df19f3ad957c0f7cf269ab23279660c941ac9873fd2ee9" exitCode=0 Jul 07 15:48:03.951450 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:03.951277 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm" Jul 07 15:48:03.951450 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:03.951335 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm" event={"ID":"17fddd64-4b62-4d34-b90d-6088bc1ae3c6","Type":"ContainerDied","Data":"ecc74922f8db98e988df19f3ad957c0f7cf269ab23279660c941ac9873fd2ee9"} Jul 07 15:48:03.951450 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:03.951377 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm" event={"ID":"17fddd64-4b62-4d34-b90d-6088bc1ae3c6","Type":"ContainerDied","Data":"33251398c1d36963d2bd2ea0817c8885e623797b38ff8329a581db5a01494482"} Jul 07 15:48:03.951450 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:03.951404 2557 scope.go:117] "RemoveContainer" containerID="ecc74922f8db98e988df19f3ad957c0f7cf269ab23279660c941ac9873fd2ee9" Jul 07 15:48:03.961852 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:03.961833 2557 scope.go:117] "RemoveContainer" containerID="ecc74922f8db98e988df19f3ad957c0f7cf269ab23279660c941ac9873fd2ee9" Jul 07 15:48:03.962140 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:48:03.962095 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecc74922f8db98e988df19f3ad957c0f7cf269ab23279660c941ac9873fd2ee9\": container with ID starting with ecc74922f8db98e988df19f3ad957c0f7cf269ab23279660c941ac9873fd2ee9 not found: ID does not exist" containerID="ecc74922f8db98e988df19f3ad957c0f7cf269ab23279660c941ac9873fd2ee9" Jul 07 15:48:03.962243 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:03.962142 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecc74922f8db98e988df19f3ad957c0f7cf269ab23279660c941ac9873fd2ee9"} err="failed to get container status \"ecc74922f8db98e988df19f3ad957c0f7cf269ab23279660c941ac9873fd2ee9\": rpc error: code = NotFound desc = could not find container \"ecc74922f8db98e988df19f3ad957c0f7cf269ab23279660c941ac9873fd2ee9\": container with ID starting with ecc74922f8db98e988df19f3ad957c0f7cf269ab23279660c941ac9873fd2ee9 not found: ID does not exist" Jul 07 15:48:03.981853 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:03.981817 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm"] Jul 07 15:48:03.984242 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:03.984219 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-q8wxm"] Jul 07 15:48:04.023154 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:04.023092 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsllm\" (UniqueName: \"kubernetes.io/projected/17fddd64-4b62-4d34-b90d-6088bc1ae3c6-kube-api-access-wsllm\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:48:04.023154 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:04.023149 2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17fddd64-4b62-4d34-b90d-6088bc1ae3c6-extensions-socket-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:48:04.352969 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:04.352925 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17fddd64-4b62-4d34-b90d-6088bc1ae3c6" path="/var/lib/kubelet/pods/17fddd64-4b62-4d34-b90d-6088bc1ae3c6/volumes" Jul 07 15:48:12.419562 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:12.419529 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-dlx9l"] Jul 07 15:48:12.420029 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:12.419917 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17fddd64-4b62-4d34-b90d-6088bc1ae3c6" containerName="manager" Jul 07 15:48:12.420029 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:12.419930 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="17fddd64-4b62-4d34-b90d-6088bc1ae3c6" containerName="manager" Jul 07 15:48:12.420029 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:12.420002 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="17fddd64-4b62-4d34-b90d-6088bc1ae3c6" containerName="manager" Jul 07 15:48:12.422053 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:12.422035 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-dlx9l" Jul 07 15:48:12.425255 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:12.425233 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rn7tr\"" Jul 07 15:48:12.432807 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:12.432784 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-dlx9l"] Jul 07 15:48:12.502243 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:12.502211 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9bnb\" (UniqueName: \"kubernetes.io/projected/aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0-kube-api-access-f9bnb\") pod \"authorino-58cd955675-dlx9l\" (UID: \"aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0\") " pod="kuadrant-system/authorino-58cd955675-dlx9l" Jul 07 15:48:12.602689 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:12.602653 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9bnb\" (UniqueName: \"kubernetes.io/projected/aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0-kube-api-access-f9bnb\") pod \"authorino-58cd955675-dlx9l\" (UID: \"aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0\") " pod="kuadrant-system/authorino-58cd955675-dlx9l" Jul 07 15:48:12.612417 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:12.612386 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9bnb\" (UniqueName: \"kubernetes.io/projected/aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0-kube-api-access-f9bnb\") pod \"authorino-58cd955675-dlx9l\" (UID: \"aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0\") " pod="kuadrant-system/authorino-58cd955675-dlx9l" Jul 07 15:48:12.732193 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:12.732100 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-dlx9l" Jul 07 15:48:12.888456 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:12.888433 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-dlx9l"] Jul 07 15:48:12.890205 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:48:12.890175 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa49f0f0_5cc2_4b90_acdc_a26ccfcc48d0.slice/crio-7b6354e03efe43c1949a6bb952235636fb233fbd3a15d59c81fe3874a3972284 WatchSource:0}: Error finding container 7b6354e03efe43c1949a6bb952235636fb233fbd3a15d59c81fe3874a3972284: Status 404 returned error can't find the container with id 7b6354e03efe43c1949a6bb952235636fb233fbd3a15d59c81fe3874a3972284 Jul 07 15:48:13.005994 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:13.005901 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-dlx9l" event={"ID":"aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0","Type":"ContainerStarted","Data":"7b6354e03efe43c1949a6bb952235636fb233fbd3a15d59c81fe3874a3972284"} Jul 07 15:48:17.038671 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:17.038631 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-dlx9l" event={"ID":"aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0","Type":"ContainerStarted","Data":"c8e96f9a72a415e5aa83ca1889674337deff8d5a38aae13d5325ac487d7adf1e"} Jul 07 15:48:17.056951 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:17.056901 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-dlx9l" podStartSLOduration=1.685279894 podStartE2EDuration="5.056886166s" totalImagesPullingTime="3.371606272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:48:17.055651848 +0000 UTC m=+481.233296298" watchObservedRunningTime="2026-07-07 15:48:17.056886166 +0000 UTC m=+481.234530616" Jul 07 15:48:17.635486 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:17.635450 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-dlx9l"] Jul 07 15:48:19.049445 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:19.049400 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-dlx9l" podUID="aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0" containerName="authorino" containerID="cri-o://c8e96f9a72a415e5aa83ca1889674337deff8d5a38aae13d5325ac487d7adf1e" gracePeriod=30 Jul 07 15:48:19.303053 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:19.302987 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-dlx9l" Jul 07 15:48:19.358722 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:19.358690 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9bnb\" (UniqueName: \"kubernetes.io/projected/aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0-kube-api-access-f9bnb\") pod \"aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0\" (UID: \"aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0\") " Jul 07 15:48:19.360763 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:19.360730 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0-kube-api-access-f9bnb" (OuterVolumeSpecName: "kube-api-access-f9bnb") pod "aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0" (UID: "aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0"). InnerVolumeSpecName "kube-api-access-f9bnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:48:19.459295 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:19.459259 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9bnb\" (UniqueName: \"kubernetes.io/projected/aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0-kube-api-access-f9bnb\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:48:20.055838 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:20.055801 2557 generic.go:356] "Generic (PLEG): container finished" podID="aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0" containerID="c8e96f9a72a415e5aa83ca1889674337deff8d5a38aae13d5325ac487d7adf1e" exitCode=0 Jul 07 15:48:20.056305 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:20.055847 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-dlx9l" event={"ID":"aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0","Type":"ContainerDied","Data":"c8e96f9a72a415e5aa83ca1889674337deff8d5a38aae13d5325ac487d7adf1e"} Jul 07 15:48:20.056305 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:20.055860 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-dlx9l" Jul 07 15:48:20.056305 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:20.055902 2557 scope.go:117] "RemoveContainer" containerID="c8e96f9a72a415e5aa83ca1889674337deff8d5a38aae13d5325ac487d7adf1e" Jul 07 15:48:20.056305 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:20.055892 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-dlx9l" event={"ID":"aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0","Type":"ContainerDied","Data":"7b6354e03efe43c1949a6bb952235636fb233fbd3a15d59c81fe3874a3972284"} Jul 07 15:48:20.064940 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:20.064915 2557 scope.go:117] "RemoveContainer" containerID="c8e96f9a72a415e5aa83ca1889674337deff8d5a38aae13d5325ac487d7adf1e" Jul 07 15:48:20.065321 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:48:20.065296 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8e96f9a72a415e5aa83ca1889674337deff8d5a38aae13d5325ac487d7adf1e\": container with ID starting with c8e96f9a72a415e5aa83ca1889674337deff8d5a38aae13d5325ac487d7adf1e not found: ID does not exist" containerID="c8e96f9a72a415e5aa83ca1889674337deff8d5a38aae13d5325ac487d7adf1e" Jul 07 15:48:20.065402 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:20.065329 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e96f9a72a415e5aa83ca1889674337deff8d5a38aae13d5325ac487d7adf1e"} err="failed to get container status \"c8e96f9a72a415e5aa83ca1889674337deff8d5a38aae13d5325ac487d7adf1e\": rpc error: code = NotFound desc = could not find container \"c8e96f9a72a415e5aa83ca1889674337deff8d5a38aae13d5325ac487d7adf1e\": container with ID starting with c8e96f9a72a415e5aa83ca1889674337deff8d5a38aae13d5325ac487d7adf1e not found: ID does not exist" Jul 07 15:48:20.077588 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:20.077560 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-dlx9l"] Jul 07 15:48:20.081768 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:20.081744 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-dlx9l"] Jul 07 15:48:20.351167 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:20.351060 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0" path="/var/lib/kubelet/pods/aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0/volumes" Jul 07 15:48:53.329638 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:53.329598 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67dfc6764d-t9s72"] Jul 07 15:48:53.330193 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:53.330045 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0" containerName="authorino" Jul 07 15:48:53.330193 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:53.330060 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0" containerName="authorino" Jul 07 15:48:53.330311 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:53.330201 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa49f0f0-5cc2-4b90-acdc-a26ccfcc48d0" containerName="authorino" Jul 07 15:48:53.337943 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:53.337918 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67dfc6764d-t9s72" Jul 07 15:48:53.340220 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:53.340192 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 07 15:48:53.340416 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:53.340280 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 07 15:48:53.340800 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:53.340784 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pgh4d\"" Jul 07 15:48:53.343601 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:53.343573 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67dfc6764d-t9s72"] Jul 07 15:48:53.463661 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:53.463617 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cmwv\" (UniqueName: \"kubernetes.io/projected/559e96e7-4836-4d6a-8dab-e3a1dc9f3221-kube-api-access-9cmwv\") pod \"maas-controller-67dfc6764d-t9s72\" (UID: \"559e96e7-4836-4d6a-8dab-e3a1dc9f3221\") " pod="opendatahub/maas-controller-67dfc6764d-t9s72" Jul 07 15:48:53.463858 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:53.463685 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/559e96e7-4836-4d6a-8dab-e3a1dc9f3221-webhook-cert\") pod \"maas-controller-67dfc6764d-t9s72\" (UID: \"559e96e7-4836-4d6a-8dab-e3a1dc9f3221\") " pod="opendatahub/maas-controller-67dfc6764d-t9s72" Jul 07 15:48:53.564956 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:53.564912 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cmwv\" (UniqueName: \"kubernetes.io/projected/559e96e7-4836-4d6a-8dab-e3a1dc9f3221-kube-api-access-9cmwv\") pod \"maas-controller-67dfc6764d-t9s72\" (UID: \"559e96e7-4836-4d6a-8dab-e3a1dc9f3221\") " pod="opendatahub/maas-controller-67dfc6764d-t9s72" Jul 07 15:48:53.564956 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:53.564956 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/559e96e7-4836-4d6a-8dab-e3a1dc9f3221-webhook-cert\") pod \"maas-controller-67dfc6764d-t9s72\" (UID: \"559e96e7-4836-4d6a-8dab-e3a1dc9f3221\") " pod="opendatahub/maas-controller-67dfc6764d-t9s72" Jul 07 15:48:53.567322 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:53.567300 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/559e96e7-4836-4d6a-8dab-e3a1dc9f3221-webhook-cert\") pod \"maas-controller-67dfc6764d-t9s72\" (UID: \"559e96e7-4836-4d6a-8dab-e3a1dc9f3221\") " pod="opendatahub/maas-controller-67dfc6764d-t9s72" Jul 07 15:48:53.574313 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:53.574286 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cmwv\" (UniqueName: \"kubernetes.io/projected/559e96e7-4836-4d6a-8dab-e3a1dc9f3221-kube-api-access-9cmwv\") pod \"maas-controller-67dfc6764d-t9s72\" (UID: \"559e96e7-4836-4d6a-8dab-e3a1dc9f3221\") " pod="opendatahub/maas-controller-67dfc6764d-t9s72" Jul 07 15:48:53.650612 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:53.650511 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67dfc6764d-t9s72" Jul 07 15:48:53.790986 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:53.790886 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67dfc6764d-t9s72"] Jul 07 15:48:53.793541 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:48:53.793514 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod559e96e7_4836_4d6a_8dab_e3a1dc9f3221.slice/crio-000666d357a8a5694ffc2a5ae066fa07d7e2777481bb884f71e2a7f297207e20 WatchSource:0}: Error finding container 000666d357a8a5694ffc2a5ae066fa07d7e2777481bb884f71e2a7f297207e20: Status 404 returned error can't find the container with id 000666d357a8a5694ffc2a5ae066fa07d7e2777481bb884f71e2a7f297207e20 Jul 07 15:48:54.252512 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:54.252478 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67dfc6764d-t9s72" event={"ID":"559e96e7-4836-4d6a-8dab-e3a1dc9f3221","Type":"ContainerStarted","Data":"000666d357a8a5694ffc2a5ae066fa07d7e2777481bb884f71e2a7f297207e20"} Jul 07 15:48:56.265706 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:56.265672 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67dfc6764d-t9s72" event={"ID":"559e96e7-4836-4d6a-8dab-e3a1dc9f3221","Type":"ContainerStarted","Data":"c4d50f3d77815729775f3dae3b53dc0ecbc282ccd4bf40c28410eff898b3b653"} Jul 07 15:48:56.266093 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:48:56.265736 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67dfc6764d-t9s72" Jul 07 15:49:07.278274 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:07.278236 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67dfc6764d-t9s72" Jul 07 15:49:07.295429 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:07.295378 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-67dfc6764d-t9s72" podStartSLOduration=12.255688648 podStartE2EDuration="14.295361643s" totalImagesPullingTime="2.039672995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:48:56.302239583 +0000 UTC m=+520.479884046" watchObservedRunningTime="2026-07-07 15:49:07.295361643 +0000 UTC m=+531.473006094" Jul 07 15:49:29.621952 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:29.621908 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-55c8bccfbb-hh996"] Jul 07 15:49:29.628259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:29.628235 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-55c8bccfbb-hh996" Jul 07 15:49:29.630567 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:29.630546 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 07 15:49:29.630985 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:29.630967 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hvc4p\"" Jul 07 15:49:29.637483 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:29.637455 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-55c8bccfbb-hh996"] Jul 07 15:49:29.684743 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:29.684710 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53121f11-e81a-4ed0-a610-78a82d907029-maas-api-tls\") pod \"maas-api-55c8bccfbb-hh996\" (UID: \"53121f11-e81a-4ed0-a610-78a82d907029\") " pod="opendatahub/maas-api-55c8bccfbb-hh996" Jul 07 15:49:29.684950 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:29.684757 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53121f11-e81a-4ed0-a610-78a82d907029-openshift-service-ca\") pod \"maas-api-55c8bccfbb-hh996\" (UID: \"53121f11-e81a-4ed0-a610-78a82d907029\") " pod="opendatahub/maas-api-55c8bccfbb-hh996" Jul 07 15:49:29.684950 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:29.684796 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mdn7\" (UniqueName: \"kubernetes.io/projected/53121f11-e81a-4ed0-a610-78a82d907029-kube-api-access-6mdn7\") pod \"maas-api-55c8bccfbb-hh996\" (UID: \"53121f11-e81a-4ed0-a610-78a82d907029\") " pod="opendatahub/maas-api-55c8bccfbb-hh996" Jul 07 15:49:29.786105 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:29.786067 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mdn7\" (UniqueName: \"kubernetes.io/projected/53121f11-e81a-4ed0-a610-78a82d907029-kube-api-access-6mdn7\") pod \"maas-api-55c8bccfbb-hh996\" (UID: \"53121f11-e81a-4ed0-a610-78a82d907029\") " pod="opendatahub/maas-api-55c8bccfbb-hh996" Jul 07 15:49:29.786346 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:29.786179 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53121f11-e81a-4ed0-a610-78a82d907029-maas-api-tls\") pod \"maas-api-55c8bccfbb-hh996\" (UID: \"53121f11-e81a-4ed0-a610-78a82d907029\") " pod="opendatahub/maas-api-55c8bccfbb-hh996" Jul 07 15:49:29.786346 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:29.786210 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53121f11-e81a-4ed0-a610-78a82d907029-openshift-service-ca\") pod \"maas-api-55c8bccfbb-hh996\" (UID: \"53121f11-e81a-4ed0-a610-78a82d907029\") " pod="opendatahub/maas-api-55c8bccfbb-hh996" Jul 07 15:49:29.786478 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:49:29.786338 2557 secret.go:189] Couldn't get secret opendatahub/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 07 15:49:29.786478 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:49:29.786429 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53121f11-e81a-4ed0-a610-78a82d907029-maas-api-tls podName:53121f11-e81a-4ed0-a610-78a82d907029 nodeName:}" failed. No retries permitted until 2026-07-07 15:49:30.286406331 +0000 UTC m=+554.464050776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/53121f11-e81a-4ed0-a610-78a82d907029-maas-api-tls") pod "maas-api-55c8bccfbb-hh996" (UID: "53121f11-e81a-4ed0-a610-78a82d907029") : secret "maas-api-serving-cert" not found Jul 07 15:49:29.786795 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:29.786776 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53121f11-e81a-4ed0-a610-78a82d907029-openshift-service-ca\") pod \"maas-api-55c8bccfbb-hh996\" (UID: \"53121f11-e81a-4ed0-a610-78a82d907029\") " pod="opendatahub/maas-api-55c8bccfbb-hh996" Jul 07 15:49:29.796100 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:29.796077 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mdn7\" (UniqueName: \"kubernetes.io/projected/53121f11-e81a-4ed0-a610-78a82d907029-kube-api-access-6mdn7\") pod \"maas-api-55c8bccfbb-hh996\" (UID: \"53121f11-e81a-4ed0-a610-78a82d907029\") " pod="opendatahub/maas-api-55c8bccfbb-hh996" Jul 07 15:49:30.174408 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:30.174370 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-xplfd"] Jul 07 15:49:30.179314 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:30.179288 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-xplfd" Jul 07 15:49:30.181784 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:30.181752 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 07 15:49:30.181932 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:30.181843 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bcvwz\"" Jul 07 15:49:30.189652 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:30.189624 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-xplfd"] Jul 07 15:49:30.292034 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:30.292002 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrfhq\" (UniqueName: \"kubernetes.io/projected/ceeafdcc-0cce-4dcb-828c-93b0965a4b54-kube-api-access-jrfhq\") pod \"payload-processing-58f86c7798-xplfd\" (UID: \"ceeafdcc-0cce-4dcb-828c-93b0965a4b54\") " pod="openshift-ingress/payload-processing-58f86c7798-xplfd" Jul 07 15:49:30.292245 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:30.292075 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceeafdcc-0cce-4dcb-828c-93b0965a4b54-plugins-config-volume\") pod \"payload-processing-58f86c7798-xplfd\" (UID: \"ceeafdcc-0cce-4dcb-828c-93b0965a4b54\") " pod="openshift-ingress/payload-processing-58f86c7798-xplfd" Jul 07 15:49:30.292245 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:30.292124 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53121f11-e81a-4ed0-a610-78a82d907029-maas-api-tls\") pod \"maas-api-55c8bccfbb-hh996\" (UID: \"53121f11-e81a-4ed0-a610-78a82d907029\") " pod="opendatahub/maas-api-55c8bccfbb-hh996" Jul 07 15:49:30.294622 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:30.294598 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53121f11-e81a-4ed0-a610-78a82d907029-maas-api-tls\") pod \"maas-api-55c8bccfbb-hh996\" (UID: \"53121f11-e81a-4ed0-a610-78a82d907029\") " pod="opendatahub/maas-api-55c8bccfbb-hh996" Jul 07 15:49:30.392991 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:30.392950 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceeafdcc-0cce-4dcb-828c-93b0965a4b54-plugins-config-volume\") pod \"payload-processing-58f86c7798-xplfd\" (UID: \"ceeafdcc-0cce-4dcb-828c-93b0965a4b54\") " pod="openshift-ingress/payload-processing-58f86c7798-xplfd" Jul 07 15:49:30.393236 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:30.393075 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrfhq\" (UniqueName: \"kubernetes.io/projected/ceeafdcc-0cce-4dcb-828c-93b0965a4b54-kube-api-access-jrfhq\") pod \"payload-processing-58f86c7798-xplfd\" (UID: \"ceeafdcc-0cce-4dcb-828c-93b0965a4b54\") " pod="openshift-ingress/payload-processing-58f86c7798-xplfd" Jul 07 15:49:30.393773 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:30.393749 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceeafdcc-0cce-4dcb-828c-93b0965a4b54-plugins-config-volume\") pod \"payload-processing-58f86c7798-xplfd\" (UID: \"ceeafdcc-0cce-4dcb-828c-93b0965a4b54\") " pod="openshift-ingress/payload-processing-58f86c7798-xplfd" Jul 07 15:49:30.401355 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:30.401333 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrfhq\" (UniqueName: \"kubernetes.io/projected/ceeafdcc-0cce-4dcb-828c-93b0965a4b54-kube-api-access-jrfhq\") pod \"payload-processing-58f86c7798-xplfd\" (UID: \"ceeafdcc-0cce-4dcb-828c-93b0965a4b54\") " pod="openshift-ingress/payload-processing-58f86c7798-xplfd" Jul 07 15:49:30.493662 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:30.493570 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-xplfd" Jul 07 15:49:30.540824 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:30.540789 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-55c8bccfbb-hh996" Jul 07 15:49:30.657679 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:30.657642 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-xplfd"] Jul 07 15:49:30.659001 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:49:30.658971 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceeafdcc_0cce_4dcb_828c_93b0965a4b54.slice/crio-4a302e504ad031608775762735e94889321c842917bd4f1b80a02ce633ea8a25 WatchSource:0}: Error finding container 4a302e504ad031608775762735e94889321c842917bd4f1b80a02ce633ea8a25: Status 404 returned error can't find the container with id 4a302e504ad031608775762735e94889321c842917bd4f1b80a02ce633ea8a25 Jul 07 15:49:30.711287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:30.711257 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-55c8bccfbb-hh996"] Jul 07 15:49:30.713296 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:49:30.713266 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53121f11_e81a_4ed0_a610_78a82d907029.slice/crio-57479410e5a84cac2f738a3665b27413ea406e79d7929c8070518eeec37ae9e8 WatchSource:0}: Error finding container 57479410e5a84cac2f738a3665b27413ea406e79d7929c8070518eeec37ae9e8: Status 404 returned error can't find the container with id 57479410e5a84cac2f738a3665b27413ea406e79d7929c8070518eeec37ae9e8 Jul 07 15:49:31.468438 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:31.468404 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-55c8bccfbb-hh996" event={"ID":"53121f11-e81a-4ed0-a610-78a82d907029","Type":"ContainerStarted","Data":"57479410e5a84cac2f738a3665b27413ea406e79d7929c8070518eeec37ae9e8"} Jul 07 15:49:31.469644 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:31.469619 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-xplfd" event={"ID":"ceeafdcc-0cce-4dcb-828c-93b0965a4b54","Type":"ContainerStarted","Data":"4a302e504ad031608775762735e94889321c842917bd4f1b80a02ce633ea8a25"} Jul 07 15:49:32.480970 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:32.480858 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-xplfd" event={"ID":"ceeafdcc-0cce-4dcb-828c-93b0965a4b54","Type":"ContainerStarted","Data":"14837210dc8a08bb7d9692d6bc6a8cf37457d4d475b0761c9fec64bdd76fc731"} Jul 07 15:49:32.481469 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:32.480995 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-xplfd" Jul 07 15:49:32.500103 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:32.500021 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-xplfd" podStartSLOduration=1.084658157 podStartE2EDuration="2.500003431s" totalImagesPullingTime="1.415345274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:49:32.497402012 +0000 UTC m=+556.675046475" watchObservedRunningTime="2026-07-07 15:49:32.500003431 +0000 UTC m=+556.677647875" Jul 07 15:49:33.488880 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:33.488841 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-55c8bccfbb-hh996" event={"ID":"53121f11-e81a-4ed0-a610-78a82d907029","Type":"ContainerStarted","Data":"dcf4ca8554a1407725e1768e291952ecc9f5e7e3aacc92df5ded1f4edf0cde01"} Jul 07 15:49:33.489407 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:33.489197 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-55c8bccfbb-hh996" Jul 07 15:49:33.507864 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:33.507808 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-55c8bccfbb-hh996" podStartSLOduration=2.621187343 podStartE2EDuration="4.507791625s" totalImagesPullingTime="1.886604282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:49:33.504846653 +0000 UTC m=+557.682491105" watchObservedRunningTime="2026-07-07 15:49:33.507791625 +0000 UTC m=+557.685436076" Jul 07 15:49:39.500390 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:39.500354 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-55c8bccfbb-hh996" Jul 07 15:49:43.490906 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:49:43.490871 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-xplfd" Jul 07 15:50:14.130125 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.130065 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8"] Jul 07 15:50:14.134327 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.134300 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:14.136695 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.136667 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 15:50:14.136841 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.136669 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 15:50:14.137284 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.137265 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jn8gz\"" Jul 07 15:50:14.137415 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.137266 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 07 15:50:14.146706 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.146679 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8"] Jul 07 15:50:14.183998 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.183959 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd5cc899-ae05-435a-8aa4-4d0044000701-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhvw8\" (UID: \"dd5cc899-ae05-435a-8aa4-4d0044000701\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:14.183998 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.184001 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd5cc899-ae05-435a-8aa4-4d0044000701-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhvw8\" (UID: \"dd5cc899-ae05-435a-8aa4-4d0044000701\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:14.184311 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.184029 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf5dj\" (UniqueName: \"kubernetes.io/projected/dd5cc899-ae05-435a-8aa4-4d0044000701-kube-api-access-xf5dj\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhvw8\" (UID: \"dd5cc899-ae05-435a-8aa4-4d0044000701\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:14.184311 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.184130 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd5cc899-ae05-435a-8aa4-4d0044000701-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhvw8\" (UID: \"dd5cc899-ae05-435a-8aa4-4d0044000701\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:14.184311 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.184224 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd5cc899-ae05-435a-8aa4-4d0044000701-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhvw8\" (UID: \"dd5cc899-ae05-435a-8aa4-4d0044000701\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:14.184311 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.184293 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd5cc899-ae05-435a-8aa4-4d0044000701-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhvw8\" (UID: \"dd5cc899-ae05-435a-8aa4-4d0044000701\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:14.285093 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.285043 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd5cc899-ae05-435a-8aa4-4d0044000701-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhvw8\" (UID: \"dd5cc899-ae05-435a-8aa4-4d0044000701\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:14.285303 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.285175 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd5cc899-ae05-435a-8aa4-4d0044000701-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhvw8\" (UID: \"dd5cc899-ae05-435a-8aa4-4d0044000701\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:14.285303 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.285206 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd5cc899-ae05-435a-8aa4-4d0044000701-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhvw8\" (UID: \"dd5cc899-ae05-435a-8aa4-4d0044000701\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:14.285303 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.285249 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf5dj\" (UniqueName: \"kubernetes.io/projected/dd5cc899-ae05-435a-8aa4-4d0044000701-kube-api-access-xf5dj\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhvw8\" (UID: \"dd5cc899-ae05-435a-8aa4-4d0044000701\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:14.285303 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.285285 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd5cc899-ae05-435a-8aa4-4d0044000701-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhvw8\" (UID: \"dd5cc899-ae05-435a-8aa4-4d0044000701\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:14.285532 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.285348 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd5cc899-ae05-435a-8aa4-4d0044000701-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhvw8\" (UID: \"dd5cc899-ae05-435a-8aa4-4d0044000701\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:14.285590 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.285557 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd5cc899-ae05-435a-8aa4-4d0044000701-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhvw8\" (UID: \"dd5cc899-ae05-435a-8aa4-4d0044000701\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:14.285635 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.285622 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd5cc899-ae05-435a-8aa4-4d0044000701-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhvw8\" (UID: \"dd5cc899-ae05-435a-8aa4-4d0044000701\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:14.285672 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.285649 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd5cc899-ae05-435a-8aa4-4d0044000701-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhvw8\" (UID: \"dd5cc899-ae05-435a-8aa4-4d0044000701\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:14.287626 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.287601 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd5cc899-ae05-435a-8aa4-4d0044000701-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhvw8\" (UID: \"dd5cc899-ae05-435a-8aa4-4d0044000701\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:14.287909 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.287891 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd5cc899-ae05-435a-8aa4-4d0044000701-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhvw8\" (UID: \"dd5cc899-ae05-435a-8aa4-4d0044000701\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:14.292401 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.292375 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf5dj\" (UniqueName: \"kubernetes.io/projected/dd5cc899-ae05-435a-8aa4-4d0044000701-kube-api-access-xf5dj\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bhvw8\" (UID: \"dd5cc899-ae05-435a-8aa4-4d0044000701\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:14.449221 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.449087 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:14.598800 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.598775 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8"] Jul 07 15:50:14.600412 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:50:14.600382 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5cc899_ae05_435a_8aa4_4d0044000701.slice/crio-06a72c49f5bb6080f7df4d8ebf9a537e4edc9ee8e7684a999717ce9cfac18683 WatchSource:0}: Error finding container 06a72c49f5bb6080f7df4d8ebf9a537e4edc9ee8e7684a999717ce9cfac18683: Status 404 returned error can't find the container with id 06a72c49f5bb6080f7df4d8ebf9a537e4edc9ee8e7684a999717ce9cfac18683 Jul 07 15:50:14.727067 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:14.726977 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" event={"ID":"dd5cc899-ae05-435a-8aa4-4d0044000701","Type":"ContainerStarted","Data":"06a72c49f5bb6080f7df4d8ebf9a537e4edc9ee8e7684a999717ce9cfac18683"} Jul 07 15:50:16.293296 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:16.293260 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/1.log" Jul 07 15:50:16.296733 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:16.296712 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/1.log" Jul 07 15:50:16.298693 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:16.298656 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hkj2z_07a80436-65b9-4f0d-99b3-ecda36240b48/ovn-acl-logging/0.log" Jul 07 15:50:16.302418 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:16.302391 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hkj2z_07a80436-65b9-4f0d-99b3-ecda36240b48/ovn-acl-logging/0.log" Jul 07 15:50:16.739778 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:16.739685 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" event={"ID":"dd5cc899-ae05-435a-8aa4-4d0044000701","Type":"ContainerStarted","Data":"6038ff4911ee9dd42a3c1fbe54ca32c39712927630174eb4288da0e43fa71adb"} Jul 07 15:50:16.759134 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:16.759057 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" podStartSLOduration=1.624424843 podStartE2EDuration="2.759036789s" totalImagesPullingTime="1.134611946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:50:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:50:16.755878984 +0000 UTC m=+600.933523454" watchObservedRunningTime="2026-07-07 15:50:16.759036789 +0000 UTC m=+600.936681239" Jul 07 15:50:24.449505 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:24.449454 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:24.452099 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:24.449548 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:24.462481 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:24.462456 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:24.797858 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:24.797825 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bhvw8" Jul 07 15:50:26.540890 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.540851 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9"] Jul 07 15:50:26.544470 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.544452 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:26.546491 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.546464 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 07 15:50:26.555589 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.555564 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9"] Jul 07 15:50:26.606919 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.606880 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c435d3d2-3245-44ba-8f34-fd99c41e8e67-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9\" (UID: \"c435d3d2-3245-44ba-8f34-fd99c41e8e67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:26.607128 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.606949 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk97l\" (UniqueName: \"kubernetes.io/projected/c435d3d2-3245-44ba-8f34-fd99c41e8e67-kube-api-access-tk97l\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9\" (UID: \"c435d3d2-3245-44ba-8f34-fd99c41e8e67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:26.607128 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.607057 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c435d3d2-3245-44ba-8f34-fd99c41e8e67-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9\" (UID: \"c435d3d2-3245-44ba-8f34-fd99c41e8e67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:26.607268 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.607196 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c435d3d2-3245-44ba-8f34-fd99c41e8e67-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9\" (UID: \"c435d3d2-3245-44ba-8f34-fd99c41e8e67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:26.607342 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.607246 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c435d3d2-3245-44ba-8f34-fd99c41e8e67-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9\" (UID: \"c435d3d2-3245-44ba-8f34-fd99c41e8e67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:26.607342 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.607317 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c435d3d2-3245-44ba-8f34-fd99c41e8e67-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9\" (UID: \"c435d3d2-3245-44ba-8f34-fd99c41e8e67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:26.708541 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.708488 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c435d3d2-3245-44ba-8f34-fd99c41e8e67-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9\" (UID: \"c435d3d2-3245-44ba-8f34-fd99c41e8e67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:26.708541 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.708560 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c435d3d2-3245-44ba-8f34-fd99c41e8e67-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9\" (UID: \"c435d3d2-3245-44ba-8f34-fd99c41e8e67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:26.708810 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.708604 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c435d3d2-3245-44ba-8f34-fd99c41e8e67-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9\" (UID: \"c435d3d2-3245-44ba-8f34-fd99c41e8e67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:26.708810 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.708647 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c435d3d2-3245-44ba-8f34-fd99c41e8e67-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9\" (UID: \"c435d3d2-3245-44ba-8f34-fd99c41e8e67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:26.708810 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.708691 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk97l\" (UniqueName: \"kubernetes.io/projected/c435d3d2-3245-44ba-8f34-fd99c41e8e67-kube-api-access-tk97l\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9\" (UID: \"c435d3d2-3245-44ba-8f34-fd99c41e8e67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:26.708810 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.708738 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c435d3d2-3245-44ba-8f34-fd99c41e8e67-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9\" (UID: \"c435d3d2-3245-44ba-8f34-fd99c41e8e67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:26.709048 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.709020 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c435d3d2-3245-44ba-8f34-fd99c41e8e67-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9\" (UID: \"c435d3d2-3245-44ba-8f34-fd99c41e8e67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:26.709170 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.709058 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c435d3d2-3245-44ba-8f34-fd99c41e8e67-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9\" (UID: \"c435d3d2-3245-44ba-8f34-fd99c41e8e67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:26.709170 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.709091 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c435d3d2-3245-44ba-8f34-fd99c41e8e67-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9\" (UID: \"c435d3d2-3245-44ba-8f34-fd99c41e8e67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:26.710966 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.710949 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c435d3d2-3245-44ba-8f34-fd99c41e8e67-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9\" (UID: \"c435d3d2-3245-44ba-8f34-fd99c41e8e67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:26.711168 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.711139 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c435d3d2-3245-44ba-8f34-fd99c41e8e67-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9\" (UID: \"c435d3d2-3245-44ba-8f34-fd99c41e8e67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:26.716623 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.716596 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk97l\" (UniqueName: \"kubernetes.io/projected/c435d3d2-3245-44ba-8f34-fd99c41e8e67-kube-api-access-tk97l\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9\" (UID: \"c435d3d2-3245-44ba-8f34-fd99c41e8e67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:26.856187 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:26.856091 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:27.003686 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:27.003654 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9"] Jul 07 15:50:27.005451 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:50:27.005427 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc435d3d2_3245_44ba_8f34_fd99c41e8e67.slice/crio-4a744a0dfb29cf80fffdaa5e6157698107d8f13ea9cbef104baf22438c2c0bcd WatchSource:0}: Error finding container 4a744a0dfb29cf80fffdaa5e6157698107d8f13ea9cbef104baf22438c2c0bcd: Status 404 returned error can't find the container with id 4a744a0dfb29cf80fffdaa5e6157698107d8f13ea9cbef104baf22438c2c0bcd Jul 07 15:50:27.007286 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:27.007267 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 15:50:27.808948 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:27.808908 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" event={"ID":"c435d3d2-3245-44ba-8f34-fd99c41e8e67","Type":"ContainerStarted","Data":"f99877a3c231f95e214d12ed146ae6edf54a5d84ccc8e9f31248ebdac91165d6"} Jul 07 15:50:27.809484 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:27.808964 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" event={"ID":"c435d3d2-3245-44ba-8f34-fd99c41e8e67","Type":"ContainerStarted","Data":"4a744a0dfb29cf80fffdaa5e6157698107d8f13ea9cbef104baf22438c2c0bcd"} Jul 07 15:50:27.828653 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:27.828600 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" podStartSLOduration=1.592032626 podStartE2EDuration="1.828584116s" totalImagesPullingTime="236.55149ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:50:27.826183112 +0000 UTC m=+612.003827588" watchObservedRunningTime="2026-07-07 15:50:27.828584116 +0000 UTC m=+612.006228565" Jul 07 15:50:36.856435 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:36.856391 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:36.856899 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:36.856458 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:36.869597 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:36.869570 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:50:37.878687 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:50:37.878652 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9" Jul 07 15:52:25.802978 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:25.802939 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-677459dbf6-zglpq"] Jul 07 15:52:25.807658 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:25.807630 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:25.820983 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:25.820956 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-677459dbf6-zglpq"] Jul 07 15:52:25.914740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:25.914705 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34dc6e32-5545-420f-a4cd-f96c249169fa-console-oauth-config\") pod \"console-677459dbf6-zglpq\" (UID: \"34dc6e32-5545-420f-a4cd-f96c249169fa\") " pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:25.914740 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:25.914742 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34dc6e32-5545-420f-a4cd-f96c249169fa-console-config\") pod \"console-677459dbf6-zglpq\" (UID: \"34dc6e32-5545-420f-a4cd-f96c249169fa\") " pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:25.914965 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:25.914773 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34dc6e32-5545-420f-a4cd-f96c249169fa-oauth-serving-cert\") pod \"console-677459dbf6-zglpq\" (UID: \"34dc6e32-5545-420f-a4cd-f96c249169fa\") " pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:25.914965 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:25.914863 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34dc6e32-5545-420f-a4cd-f96c249169fa-trusted-ca-bundle\") pod \"console-677459dbf6-zglpq\" (UID: \"34dc6e32-5545-420f-a4cd-f96c249169fa\") " pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:25.914965 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:25.914904 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqgmj\" (UniqueName: \"kubernetes.io/projected/34dc6e32-5545-420f-a4cd-f96c249169fa-kube-api-access-rqgmj\") pod \"console-677459dbf6-zglpq\" (UID: \"34dc6e32-5545-420f-a4cd-f96c249169fa\") " pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:25.914965 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:25.914924 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34dc6e32-5545-420f-a4cd-f96c249169fa-console-serving-cert\") pod \"console-677459dbf6-zglpq\" (UID: \"34dc6e32-5545-420f-a4cd-f96c249169fa\") " pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:25.915187 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:25.914978 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34dc6e32-5545-420f-a4cd-f96c249169fa-service-ca\") pod \"console-677459dbf6-zglpq\" (UID: \"34dc6e32-5545-420f-a4cd-f96c249169fa\") " pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:26.016126 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:26.016070 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqgmj\" (UniqueName: \"kubernetes.io/projected/34dc6e32-5545-420f-a4cd-f96c249169fa-kube-api-access-rqgmj\") pod \"console-677459dbf6-zglpq\" (UID: \"34dc6e32-5545-420f-a4cd-f96c249169fa\") " pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:26.016335 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:26.016139 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34dc6e32-5545-420f-a4cd-f96c249169fa-console-serving-cert\") pod \"console-677459dbf6-zglpq\" (UID: \"34dc6e32-5545-420f-a4cd-f96c249169fa\") " pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:26.016335 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:26.016206 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34dc6e32-5545-420f-a4cd-f96c249169fa-service-ca\") pod \"console-677459dbf6-zglpq\" (UID: \"34dc6e32-5545-420f-a4cd-f96c249169fa\") " pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:26.016335 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:26.016274 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34dc6e32-5545-420f-a4cd-f96c249169fa-console-oauth-config\") pod \"console-677459dbf6-zglpq\" (UID: \"34dc6e32-5545-420f-a4cd-f96c249169fa\") " pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:26.016335 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:26.016310 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34dc6e32-5545-420f-a4cd-f96c249169fa-console-config\") pod \"console-677459dbf6-zglpq\" (UID: \"34dc6e32-5545-420f-a4cd-f96c249169fa\") " pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:26.016569 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:26.016363 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34dc6e32-5545-420f-a4cd-f96c249169fa-oauth-serving-cert\") pod \"console-677459dbf6-zglpq\" (UID: \"34dc6e32-5545-420f-a4cd-f96c249169fa\") " pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:26.016569 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:26.016410 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34dc6e32-5545-420f-a4cd-f96c249169fa-trusted-ca-bundle\") pod \"console-677459dbf6-zglpq\" (UID: \"34dc6e32-5545-420f-a4cd-f96c249169fa\") " pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:26.017103 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:26.017077 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34dc6e32-5545-420f-a4cd-f96c249169fa-oauth-serving-cert\") pod \"console-677459dbf6-zglpq\" (UID: \"34dc6e32-5545-420f-a4cd-f96c249169fa\") " pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:26.017268 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:26.017153 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34dc6e32-5545-420f-a4cd-f96c249169fa-console-config\") pod \"console-677459dbf6-zglpq\" (UID: \"34dc6e32-5545-420f-a4cd-f96c249169fa\") " pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:26.017268 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:26.017183 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34dc6e32-5545-420f-a4cd-f96c249169fa-service-ca\") pod \"console-677459dbf6-zglpq\" (UID: \"34dc6e32-5545-420f-a4cd-f96c249169fa\") " pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:26.017382 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:26.017314 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34dc6e32-5545-420f-a4cd-f96c249169fa-trusted-ca-bundle\") pod \"console-677459dbf6-zglpq\" (UID: \"34dc6e32-5545-420f-a4cd-f96c249169fa\") " pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:26.019259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:26.019236 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34dc6e32-5545-420f-a4cd-f96c249169fa-console-serving-cert\") pod \"console-677459dbf6-zglpq\" (UID: \"34dc6e32-5545-420f-a4cd-f96c249169fa\") " pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:26.019259 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:26.019245 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34dc6e32-5545-420f-a4cd-f96c249169fa-console-oauth-config\") pod \"console-677459dbf6-zglpq\" (UID: \"34dc6e32-5545-420f-a4cd-f96c249169fa\") " pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:26.024557 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:26.024537 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqgmj\" (UniqueName: \"kubernetes.io/projected/34dc6e32-5545-420f-a4cd-f96c249169fa-kube-api-access-rqgmj\") pod \"console-677459dbf6-zglpq\" (UID: \"34dc6e32-5545-420f-a4cd-f96c249169fa\") " pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:26.117906 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:26.117807 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:26.270026 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:26.269997 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-677459dbf6-zglpq"] Jul 07 15:52:26.272194 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:52:26.272157 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34dc6e32_5545_420f_a4cd_f96c249169fa.slice/crio-e9fe14e791c37db30a3a32ca8a37b9953b85a45c555d0e97ca583f53dd640884 WatchSource:0}: Error finding container e9fe14e791c37db30a3a32ca8a37b9953b85a45c555d0e97ca583f53dd640884: Status 404 returned error can't find the container with id e9fe14e791c37db30a3a32ca8a37b9953b85a45c555d0e97ca583f53dd640884 Jul 07 15:52:26.509948 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:26.509841 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-677459dbf6-zglpq" event={"ID":"34dc6e32-5545-420f-a4cd-f96c249169fa","Type":"ContainerStarted","Data":"9816d9dd6910fc0a7244a9051296094df7686253c56be03fe500eb51477a78eb"} Jul 07 15:52:26.509948 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:26.509895 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-677459dbf6-zglpq" event={"ID":"34dc6e32-5545-420f-a4cd-f96c249169fa","Type":"ContainerStarted","Data":"e9fe14e791c37db30a3a32ca8a37b9953b85a45c555d0e97ca583f53dd640884"} Jul 07 15:52:26.529106 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:26.529006 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-677459dbf6-zglpq" podStartSLOduration=1.528989101 podStartE2EDuration="1.528989101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:52:26.528592951 +0000 UTC m=+730.706237420" watchObservedRunningTime="2026-07-07 15:52:26.528989101 +0000 UTC m=+730.706633552" Jul 07 15:52:36.117959 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:36.117918 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:36.118521 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:36.118001 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:36.122934 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:36.122908 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:36.572159 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:36.572104 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-677459dbf6-zglpq" Jul 07 15:52:36.619019 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:52:36.618984 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-787c787b88-5t5fn"] Jul 07 15:53:01.648476 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:01.648433 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-787c787b88-5t5fn" podUID="8419d369-c539-4f2e-9eb2-caaaf639d23b" containerName="console" containerID="cri-o://520ea0496d0251d5fe58f4d8484862eb7a4a8339ebd2ae1b1f780668f53e6474" gracePeriod=15 Jul 07 15:53:01.917508 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:01.917481 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:53:02.061027 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.060988 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-oauth-serving-cert\") pod \"8419d369-c539-4f2e-9eb2-caaaf639d23b\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " Jul 07 15:53:02.061027 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.061037 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-trusted-ca-bundle\") pod \"8419d369-c539-4f2e-9eb2-caaaf639d23b\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " Jul 07 15:53:02.061287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.061091 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc2cx\" (UniqueName: \"kubernetes.io/projected/8419d369-c539-4f2e-9eb2-caaaf639d23b-kube-api-access-lc2cx\") pod \"8419d369-c539-4f2e-9eb2-caaaf639d23b\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " Jul 07 15:53:02.061287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.061167 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-console-config\") pod \"8419d369-c539-4f2e-9eb2-caaaf639d23b\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " Jul 07 15:53:02.061287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.061216 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8419d369-c539-4f2e-9eb2-caaaf639d23b-console-oauth-config\") pod \"8419d369-c539-4f2e-9eb2-caaaf639d23b\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " Jul 07 15:53:02.061287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.061259 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-service-ca\") pod \"8419d369-c539-4f2e-9eb2-caaaf639d23b\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " Jul 07 15:53:02.061287 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.061283 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8419d369-c539-4f2e-9eb2-caaaf639d23b-console-serving-cert\") pod \"8419d369-c539-4f2e-9eb2-caaaf639d23b\" (UID: \"8419d369-c539-4f2e-9eb2-caaaf639d23b\") " Jul 07 15:53:02.061662 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.061631 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8419d369-c539-4f2e-9eb2-caaaf639d23b" (UID: "8419d369-c539-4f2e-9eb2-caaaf639d23b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:53:02.061662 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.061479 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8419d369-c539-4f2e-9eb2-caaaf639d23b" (UID: "8419d369-c539-4f2e-9eb2-caaaf639d23b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:53:02.061864 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.061684 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-service-ca" (OuterVolumeSpecName: "service-ca") pod "8419d369-c539-4f2e-9eb2-caaaf639d23b" (UID: "8419d369-c539-4f2e-9eb2-caaaf639d23b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:53:02.061864 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.061850 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-console-config" (OuterVolumeSpecName: "console-config") pod "8419d369-c539-4f2e-9eb2-caaaf639d23b" (UID: "8419d369-c539-4f2e-9eb2-caaaf639d23b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:53:02.063468 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.063448 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8419d369-c539-4f2e-9eb2-caaaf639d23b-kube-api-access-lc2cx" (OuterVolumeSpecName: "kube-api-access-lc2cx") pod "8419d369-c539-4f2e-9eb2-caaaf639d23b" (UID: "8419d369-c539-4f2e-9eb2-caaaf639d23b"). InnerVolumeSpecName "kube-api-access-lc2cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:53:02.063551 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.063480 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8419d369-c539-4f2e-9eb2-caaaf639d23b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8419d369-c539-4f2e-9eb2-caaaf639d23b" (UID: "8419d369-c539-4f2e-9eb2-caaaf639d23b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:53:02.063639 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.063619 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8419d369-c539-4f2e-9eb2-caaaf639d23b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8419d369-c539-4f2e-9eb2-caaaf639d23b" (UID: "8419d369-c539-4f2e-9eb2-caaaf639d23b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:53:02.162078 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.161981 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-console-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:53:02.162078 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.162013 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8419d369-c539-4f2e-9eb2-caaaf639d23b-console-oauth-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:53:02.162078 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.162026 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:53:02.162078 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.162035 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8419d369-c539-4f2e-9eb2-caaaf639d23b-console-serving-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:53:02.162078 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.162044 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-oauth-serving-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:53:02.162078 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.162052 2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8419d369-c539-4f2e-9eb2-caaaf639d23b-trusted-ca-bundle\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:53:02.162078 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.162060 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc2cx\" (UniqueName: \"kubernetes.io/projected/8419d369-c539-4f2e-9eb2-caaaf639d23b-kube-api-access-lc2cx\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:53:02.731790 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.731759 2557 generic.go:356] "Generic (PLEG): container finished" podID="8419d369-c539-4f2e-9eb2-caaaf639d23b" containerID="520ea0496d0251d5fe58f4d8484862eb7a4a8339ebd2ae1b1f780668f53e6474" exitCode=0 Jul 07 15:53:02.732270 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.731800 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-787c787b88-5t5fn" Jul 07 15:53:02.732270 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.731813 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-787c787b88-5t5fn" event={"ID":"8419d369-c539-4f2e-9eb2-caaaf639d23b","Type":"ContainerDied","Data":"520ea0496d0251d5fe58f4d8484862eb7a4a8339ebd2ae1b1f780668f53e6474"} Jul 07 15:53:02.732270 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.731855 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-787c787b88-5t5fn" event={"ID":"8419d369-c539-4f2e-9eb2-caaaf639d23b","Type":"ContainerDied","Data":"cc90d194dfca9106a32e90fd19ff01a998dec908c8c312d7283886fb0c0f29de"} Jul 07 15:53:02.732270 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.731876 2557 scope.go:117] "RemoveContainer" containerID="520ea0496d0251d5fe58f4d8484862eb7a4a8339ebd2ae1b1f780668f53e6474" Jul 07 15:53:02.741279 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.741257 2557 scope.go:117] "RemoveContainer" containerID="520ea0496d0251d5fe58f4d8484862eb7a4a8339ebd2ae1b1f780668f53e6474" Jul 07 15:53:02.741545 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:53:02.741525 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"520ea0496d0251d5fe58f4d8484862eb7a4a8339ebd2ae1b1f780668f53e6474\": container with ID starting with 520ea0496d0251d5fe58f4d8484862eb7a4a8339ebd2ae1b1f780668f53e6474 not found: ID does not exist" containerID="520ea0496d0251d5fe58f4d8484862eb7a4a8339ebd2ae1b1f780668f53e6474" Jul 07 15:53:02.741614 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.741554 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520ea0496d0251d5fe58f4d8484862eb7a4a8339ebd2ae1b1f780668f53e6474"} err="failed to get container status \"520ea0496d0251d5fe58f4d8484862eb7a4a8339ebd2ae1b1f780668f53e6474\": rpc error: code = NotFound desc = could not find container \"520ea0496d0251d5fe58f4d8484862eb7a4a8339ebd2ae1b1f780668f53e6474\": container with ID starting with 520ea0496d0251d5fe58f4d8484862eb7a4a8339ebd2ae1b1f780668f53e6474 not found: ID does not exist" Jul 07 15:53:02.748071 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.748044 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-787c787b88-5t5fn"] Jul 07 15:53:02.752470 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:02.752448 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-787c787b88-5t5fn"] Jul 07 15:53:04.352655 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:04.352623 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8419d369-c539-4f2e-9eb2-caaaf639d23b" path="/var/lib/kubelet/pods/8419d369-c539-4f2e-9eb2-caaaf639d23b/volumes" Jul 07 15:53:05.083850 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:05.083810 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67dfc6764d-t9s72"] Jul 07 15:53:05.084101 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:05.084075 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-67dfc6764d-t9s72" podUID="559e96e7-4836-4d6a-8dab-e3a1dc9f3221" containerName="manager" containerID="cri-o://c4d50f3d77815729775f3dae3b53dc0ecbc282ccd4bf40c28410eff898b3b653" gracePeriod=30 Jul 07 15:53:05.350238 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:05.350168 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67dfc6764d-t9s72" Jul 07 15:53:05.496016 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:05.495976 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/559e96e7-4836-4d6a-8dab-e3a1dc9f3221-webhook-cert\") pod \"559e96e7-4836-4d6a-8dab-e3a1dc9f3221\" (UID: \"559e96e7-4836-4d6a-8dab-e3a1dc9f3221\") " Jul 07 15:53:05.496524 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:05.496037 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cmwv\" (UniqueName: \"kubernetes.io/projected/559e96e7-4836-4d6a-8dab-e3a1dc9f3221-kube-api-access-9cmwv\") pod \"559e96e7-4836-4d6a-8dab-e3a1dc9f3221\" (UID: \"559e96e7-4836-4d6a-8dab-e3a1dc9f3221\") " Jul 07 15:53:05.498378 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:05.498341 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/559e96e7-4836-4d6a-8dab-e3a1dc9f3221-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "559e96e7-4836-4d6a-8dab-e3a1dc9f3221" (UID: "559e96e7-4836-4d6a-8dab-e3a1dc9f3221"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:53:05.498502 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:05.498383 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/559e96e7-4836-4d6a-8dab-e3a1dc9f3221-kube-api-access-9cmwv" (OuterVolumeSpecName: "kube-api-access-9cmwv") pod "559e96e7-4836-4d6a-8dab-e3a1dc9f3221" (UID: "559e96e7-4836-4d6a-8dab-e3a1dc9f3221"). InnerVolumeSpecName "kube-api-access-9cmwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:53:05.597668 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:05.597625 2557 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/559e96e7-4836-4d6a-8dab-e3a1dc9f3221-webhook-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:53:05.597668 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:05.597662 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cmwv\" (UniqueName: \"kubernetes.io/projected/559e96e7-4836-4d6a-8dab-e3a1dc9f3221-kube-api-access-9cmwv\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:53:05.750792 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:05.750697 2557 generic.go:356] "Generic (PLEG): container finished" podID="559e96e7-4836-4d6a-8dab-e3a1dc9f3221" containerID="c4d50f3d77815729775f3dae3b53dc0ecbc282ccd4bf40c28410eff898b3b653" exitCode=0 Jul 07 15:53:05.750792 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:05.750746 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67dfc6764d-t9s72" Jul 07 15:53:05.750792 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:05.750761 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67dfc6764d-t9s72" event={"ID":"559e96e7-4836-4d6a-8dab-e3a1dc9f3221","Type":"ContainerDied","Data":"c4d50f3d77815729775f3dae3b53dc0ecbc282ccd4bf40c28410eff898b3b653"} Jul 07 15:53:05.751043 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:05.750818 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67dfc6764d-t9s72" event={"ID":"559e96e7-4836-4d6a-8dab-e3a1dc9f3221","Type":"ContainerDied","Data":"000666d357a8a5694ffc2a5ae066fa07d7e2777481bb884f71e2a7f297207e20"} Jul 07 15:53:05.751043 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:05.750849 2557 scope.go:117] "RemoveContainer" containerID="c4d50f3d77815729775f3dae3b53dc0ecbc282ccd4bf40c28410eff898b3b653" Jul 07 15:53:05.761235 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:05.761210 2557 scope.go:117] "RemoveContainer" containerID="c4d50f3d77815729775f3dae3b53dc0ecbc282ccd4bf40c28410eff898b3b653" Jul 07 15:53:05.761521 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:53:05.761500 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d50f3d77815729775f3dae3b53dc0ecbc282ccd4bf40c28410eff898b3b653\": container with ID starting with c4d50f3d77815729775f3dae3b53dc0ecbc282ccd4bf40c28410eff898b3b653 not found: ID does not exist" containerID="c4d50f3d77815729775f3dae3b53dc0ecbc282ccd4bf40c28410eff898b3b653" Jul 07 15:53:05.761590 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:05.761530 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d50f3d77815729775f3dae3b53dc0ecbc282ccd4bf40c28410eff898b3b653"} err="failed to get container status \"c4d50f3d77815729775f3dae3b53dc0ecbc282ccd4bf40c28410eff898b3b653\": rpc error: code = NotFound desc = could not find container \"c4d50f3d77815729775f3dae3b53dc0ecbc282ccd4bf40c28410eff898b3b653\": container with ID starting with c4d50f3d77815729775f3dae3b53dc0ecbc282ccd4bf40c28410eff898b3b653 not found: ID does not exist" Jul 07 15:53:05.772689 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:05.772659 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67dfc6764d-t9s72"] Jul 07 15:53:05.779015 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:05.778986 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-67dfc6764d-t9s72"] Jul 07 15:53:06.352189 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:06.352153 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="559e96e7-4836-4d6a-8dab-e3a1dc9f3221" path="/var/lib/kubelet/pods/559e96e7-4836-4d6a-8dab-e3a1dc9f3221/volumes" Jul 07 15:53:06.549830 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:06.549794 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67dfc6764d-dbhsj"] Jul 07 15:53:06.550253 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:06.550239 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="559e96e7-4836-4d6a-8dab-e3a1dc9f3221" containerName="manager" Jul 07 15:53:06.550253 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:06.550254 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="559e96e7-4836-4d6a-8dab-e3a1dc9f3221" containerName="manager" Jul 07 15:53:06.550371 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:06.550266 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8419d369-c539-4f2e-9eb2-caaaf639d23b" containerName="console" Jul 07 15:53:06.550371 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:06.550271 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8419d369-c539-4f2e-9eb2-caaaf639d23b" containerName="console" Jul 07 15:53:06.550371 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:06.550358 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8419d369-c539-4f2e-9eb2-caaaf639d23b" containerName="console" Jul 07 15:53:06.550371 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:06.550368 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="559e96e7-4836-4d6a-8dab-e3a1dc9f3221" containerName="manager" Jul 07 15:53:06.554686 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:06.554664 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67dfc6764d-dbhsj" Jul 07 15:53:06.556717 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:06.556694 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 07 15:53:06.557118 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:06.557094 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pgh4d\"" Jul 07 15:53:06.557180 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:06.557165 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 07 15:53:06.563879 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:06.563850 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67dfc6764d-dbhsj"] Jul 07 15:53:06.706904 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:06.706821 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbrzb\" (UniqueName: \"kubernetes.io/projected/6ef0c63c-e464-461a-9933-bae62726907d-kube-api-access-wbrzb\") pod \"maas-controller-67dfc6764d-dbhsj\" (UID: \"6ef0c63c-e464-461a-9933-bae62726907d\") " pod="opendatahub/maas-controller-67dfc6764d-dbhsj" Jul 07 15:53:06.706904 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:06.706857 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ef0c63c-e464-461a-9933-bae62726907d-webhook-cert\") pod \"maas-controller-67dfc6764d-dbhsj\" (UID: \"6ef0c63c-e464-461a-9933-bae62726907d\") " pod="opendatahub/maas-controller-67dfc6764d-dbhsj" Jul 07 15:53:06.807956 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:06.807916 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbrzb\" (UniqueName: \"kubernetes.io/projected/6ef0c63c-e464-461a-9933-bae62726907d-kube-api-access-wbrzb\") pod \"maas-controller-67dfc6764d-dbhsj\" (UID: \"6ef0c63c-e464-461a-9933-bae62726907d\") " pod="opendatahub/maas-controller-67dfc6764d-dbhsj" Jul 07 15:53:06.807956 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:06.807956 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ef0c63c-e464-461a-9933-bae62726907d-webhook-cert\") pod \"maas-controller-67dfc6764d-dbhsj\" (UID: \"6ef0c63c-e464-461a-9933-bae62726907d\") " pod="opendatahub/maas-controller-67dfc6764d-dbhsj" Jul 07 15:53:06.810529 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:06.810506 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ef0c63c-e464-461a-9933-bae62726907d-webhook-cert\") pod \"maas-controller-67dfc6764d-dbhsj\" (UID: \"6ef0c63c-e464-461a-9933-bae62726907d\") " pod="opendatahub/maas-controller-67dfc6764d-dbhsj" Jul 07 15:53:06.816020 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:06.815993 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbrzb\" (UniqueName: \"kubernetes.io/projected/6ef0c63c-e464-461a-9933-bae62726907d-kube-api-access-wbrzb\") pod \"maas-controller-67dfc6764d-dbhsj\" (UID: \"6ef0c63c-e464-461a-9933-bae62726907d\") " pod="opendatahub/maas-controller-67dfc6764d-dbhsj" Jul 07 15:53:06.866714 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:06.866678 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67dfc6764d-dbhsj" Jul 07 15:53:07.016244 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:07.016215 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67dfc6764d-dbhsj"] Jul 07 15:53:07.019040 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:53:07.019008 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef0c63c_e464_461a_9933_bae62726907d.slice/crio-fc5a9c79fb481203c190508af01a88b752de368ea8bf1a1f1d0c2f530bbcfcac WatchSource:0}: Error finding container fc5a9c79fb481203c190508af01a88b752de368ea8bf1a1f1d0c2f530bbcfcac: Status 404 returned error can't find the container with id fc5a9c79fb481203c190508af01a88b752de368ea8bf1a1f1d0c2f530bbcfcac Jul 07 15:53:07.764937 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:07.764899 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67dfc6764d-dbhsj" event={"ID":"6ef0c63c-e464-461a-9933-bae62726907d","Type":"ContainerStarted","Data":"48cbfd7a71c302cc6f3bf6cda1d4b5f1939061f71cefe8875a0eb4fb0ebbd115"} Jul 07 15:53:07.765399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:07.764968 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67dfc6764d-dbhsj" event={"ID":"6ef0c63c-e464-461a-9933-bae62726907d","Type":"ContainerStarted","Data":"fc5a9c79fb481203c190508af01a88b752de368ea8bf1a1f1d0c2f530bbcfcac"} Jul 07 15:53:07.765399 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:07.765089 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67dfc6764d-dbhsj" Jul 07 15:53:07.783390 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:07.783314 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-67dfc6764d-dbhsj" podStartSLOduration=1.193057949 podStartE2EDuration="1.783292516s" totalImagesPullingTime="590.234567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:53:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:53:07.781185167 +0000 UTC m=+771.958829616" watchObservedRunningTime="2026-07-07 15:53:07.783292516 +0000 UTC m=+771.960936967" Jul 07 15:53:18.776989 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:53:18.776956 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67dfc6764d-dbhsj" Jul 07 15:55:16.337778 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:55:16.337702 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/1.log" Jul 07 15:55:16.337778 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:55:16.337702 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/1.log" Jul 07 15:55:16.343009 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:55:16.342989 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hkj2z_07a80436-65b9-4f0d-99b3-ecda36240b48/ovn-acl-logging/0.log" Jul 07 15:55:16.343169 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:55:16.343020 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hkj2z_07a80436-65b9-4f0d-99b3-ecda36240b48/ovn-acl-logging/0.log" Jul 07 15:57:56.480467 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:57:56.480426 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm"] Jul 07 15:57:56.481085 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:57:56.480664 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm" podUID="c45aca8d-b099-4efa-a2bb-1f5138cc072b" containerName="manager" containerID="cri-o://62917a4d9c8f935b9bb9c1d4381fb46a9e1bd8b9449dbb0c793678e291f3d18b" gracePeriod=10 Jul 07 15:58:00.437080 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:00.437045 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm" Jul 07 15:58:00.478424 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:00.478391 2557 generic.go:356] "Generic (PLEG): container finished" podID="c45aca8d-b099-4efa-a2bb-1f5138cc072b" containerID="62917a4d9c8f935b9bb9c1d4381fb46a9e1bd8b9449dbb0c793678e291f3d18b" exitCode=0 Jul 07 15:58:00.478620 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:00.478435 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm" Jul 07 15:58:00.478620 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:00.478470 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm" event={"ID":"c45aca8d-b099-4efa-a2bb-1f5138cc072b","Type":"ContainerDied","Data":"62917a4d9c8f935b9bb9c1d4381fb46a9e1bd8b9449dbb0c793678e291f3d18b"} Jul 07 15:58:00.478620 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:00.478535 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm" event={"ID":"c45aca8d-b099-4efa-a2bb-1f5138cc072b","Type":"ContainerDied","Data":"8eab9efcb0b8e5df314747c7cc279f34e73aaae085e673395ae0193472f7f2f1"} Jul 07 15:58:00.478620 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:00.478564 2557 scope.go:117] "RemoveContainer" containerID="62917a4d9c8f935b9bb9c1d4381fb46a9e1bd8b9449dbb0c793678e291f3d18b" Jul 07 15:58:00.489964 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:00.489942 2557 scope.go:117] "RemoveContainer" containerID="62917a4d9c8f935b9bb9c1d4381fb46a9e1bd8b9449dbb0c793678e291f3d18b" Jul 07 15:58:00.490307 ip-10-0-139-248 kubenswrapper[2557]: E0707 15:58:00.490284 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62917a4d9c8f935b9bb9c1d4381fb46a9e1bd8b9449dbb0c793678e291f3d18b\": container with ID starting with 62917a4d9c8f935b9bb9c1d4381fb46a9e1bd8b9449dbb0c793678e291f3d18b not found: ID does not exist" containerID="62917a4d9c8f935b9bb9c1d4381fb46a9e1bd8b9449dbb0c793678e291f3d18b" Jul 07 15:58:00.490393 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:00.490317 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62917a4d9c8f935b9bb9c1d4381fb46a9e1bd8b9449dbb0c793678e291f3d18b"} err="failed to get container status \"62917a4d9c8f935b9bb9c1d4381fb46a9e1bd8b9449dbb0c793678e291f3d18b\": rpc error: code = NotFound desc = could not find container \"62917a4d9c8f935b9bb9c1d4381fb46a9e1bd8b9449dbb0c793678e291f3d18b\": container with ID starting with 62917a4d9c8f935b9bb9c1d4381fb46a9e1bd8b9449dbb0c793678e291f3d18b not found: ID does not exist" Jul 07 15:58:00.606768 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:00.606678 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72pjn\" (UniqueName: \"kubernetes.io/projected/c45aca8d-b099-4efa-a2bb-1f5138cc072b-kube-api-access-72pjn\") pod \"c45aca8d-b099-4efa-a2bb-1f5138cc072b\" (UID: \"c45aca8d-b099-4efa-a2bb-1f5138cc072b\") " Jul 07 15:58:00.606768 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:00.606725 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c45aca8d-b099-4efa-a2bb-1f5138cc072b-extensions-socket-volume\") pod \"c45aca8d-b099-4efa-a2bb-1f5138cc072b\" (UID: \"c45aca8d-b099-4efa-a2bb-1f5138cc072b\") " Jul 07 15:58:00.607142 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:00.607095 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45aca8d-b099-4efa-a2bb-1f5138cc072b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c45aca8d-b099-4efa-a2bb-1f5138cc072b" (UID: "c45aca8d-b099-4efa-a2bb-1f5138cc072b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:58:00.608861 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:00.608831 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45aca8d-b099-4efa-a2bb-1f5138cc072b-kube-api-access-72pjn" (OuterVolumeSpecName: "kube-api-access-72pjn") pod "c45aca8d-b099-4efa-a2bb-1f5138cc072b" (UID: "c45aca8d-b099-4efa-a2bb-1f5138cc072b"). InnerVolumeSpecName "kube-api-access-72pjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:58:00.707790 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:00.707750 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72pjn\" (UniqueName: \"kubernetes.io/projected/c45aca8d-b099-4efa-a2bb-1f5138cc072b-kube-api-access-72pjn\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:58:00.707790 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:00.707782 2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c45aca8d-b099-4efa-a2bb-1f5138cc072b-extensions-socket-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 15:58:00.800439 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:00.800396 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm"] Jul 07 15:58:00.804199 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:00.804173 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5q2qm"] Jul 07 15:58:02.351768 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:02.351726 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c45aca8d-b099-4efa-a2bb-1f5138cc072b" path="/var/lib/kubelet/pods/c45aca8d-b099-4efa-a2bb-1f5138cc072b/volumes" Jul 07 15:58:57.669260 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:57.669217 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bgcld"] Jul 07 15:58:57.669753 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:57.669658 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c45aca8d-b099-4efa-a2bb-1f5138cc072b" containerName="manager" Jul 07 15:58:57.669753 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:57.669697 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c45aca8d-b099-4efa-a2bb-1f5138cc072b" containerName="manager" Jul 07 15:58:57.669834 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:57.669802 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c45aca8d-b099-4efa-a2bb-1f5138cc072b" containerName="manager" Jul 07 15:58:57.673083 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:57.673068 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bgcld" Jul 07 15:58:57.676281 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:57.676261 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bh7bf\"" Jul 07 15:58:57.702232 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:57.702203 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bgcld"] Jul 07 15:58:57.829133 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:57.829079 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dlgl\" (UniqueName: \"kubernetes.io/projected/670b1e18-e596-4558-8bba-123a1f17d805-kube-api-access-6dlgl\") pod \"kuadrant-operator-controller-manager-86878c769f-bgcld\" (UID: \"670b1e18-e596-4558-8bba-123a1f17d805\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bgcld" Jul 07 15:58:57.829332 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:57.829215 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/670b1e18-e596-4558-8bba-123a1f17d805-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bgcld\" (UID: \"670b1e18-e596-4558-8bba-123a1f17d805\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bgcld" Jul 07 15:58:57.930744 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:57.930642 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dlgl\" (UniqueName: \"kubernetes.io/projected/670b1e18-e596-4558-8bba-123a1f17d805-kube-api-access-6dlgl\") pod \"kuadrant-operator-controller-manager-86878c769f-bgcld\" (UID: \"670b1e18-e596-4558-8bba-123a1f17d805\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bgcld" Jul 07 15:58:57.930744 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:57.930708 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/670b1e18-e596-4558-8bba-123a1f17d805-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bgcld\" (UID: \"670b1e18-e596-4558-8bba-123a1f17d805\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bgcld" Jul 07 15:58:57.931057 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:57.931040 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/670b1e18-e596-4558-8bba-123a1f17d805-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bgcld\" (UID: \"670b1e18-e596-4558-8bba-123a1f17d805\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bgcld" Jul 07 15:58:57.941104 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:57.941075 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dlgl\" (UniqueName: \"kubernetes.io/projected/670b1e18-e596-4558-8bba-123a1f17d805-kube-api-access-6dlgl\") pod \"kuadrant-operator-controller-manager-86878c769f-bgcld\" (UID: \"670b1e18-e596-4558-8bba-123a1f17d805\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bgcld" Jul 07 15:58:57.983594 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:57.983569 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bgcld" Jul 07 15:58:58.132346 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:58.132301 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bgcld"] Jul 07 15:58:58.133740 ip-10-0-139-248 kubenswrapper[2557]: W0707 15:58:58.133708 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670b1e18_e596_4558_8bba_123a1f17d805.slice/crio-ee17b7f0afbd46b708a863c0c9682ffc8d364a7a61c4a1f6515232d59d95d14a WatchSource:0}: Error finding container ee17b7f0afbd46b708a863c0c9682ffc8d364a7a61c4a1f6515232d59d95d14a: Status 404 returned error can't find the container with id ee17b7f0afbd46b708a863c0c9682ffc8d364a7a61c4a1f6515232d59d95d14a Jul 07 15:58:58.818894 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:58.818848 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bgcld" event={"ID":"670b1e18-e596-4558-8bba-123a1f17d805","Type":"ContainerStarted","Data":"a90efecc018d0fbaf746968ee83fc15e5e23a7bf425dfa31971799ae2df814be"} Jul 07 15:58:58.819312 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:58.818910 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bgcld" Jul 07 15:58:58.819312 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:58.818926 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bgcld" event={"ID":"670b1e18-e596-4558-8bba-123a1f17d805","Type":"ContainerStarted","Data":"ee17b7f0afbd46b708a863c0c9682ffc8d364a7a61c4a1f6515232d59d95d14a"} Jul 07 15:58:58.838109 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:58:58.838052 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bgcld" podStartSLOduration=1.838032692 podStartE2EDuration="1.838032692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:58:58.837461574 +0000 UTC m=+1123.015106050" watchObservedRunningTime="2026-07-07 15:58:58.838032692 +0000 UTC m=+1123.015677143" Jul 07 15:59:09.826677 ip-10-0-139-248 kubenswrapper[2557]: I0707 15:59:09.826643 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bgcld" Jul 07 16:00:00.148196 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:00.148095 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29724000-p7lq6"] Jul 07 16:00:00.151990 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:00.151970 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724000-p7lq6" Jul 07 16:00:00.161055 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:00.161032 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724000-p7lq6"] Jul 07 16:00:00.203123 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:00.203087 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9446\" (UniqueName: \"kubernetes.io/projected/7b16651f-b2cb-4e87-bc5a-3684c373e07a-kube-api-access-b9446\") pod \"maas-api-key-cleanup-29724000-p7lq6\" (UID: \"7b16651f-b2cb-4e87-bc5a-3684c373e07a\") " pod="opendatahub/maas-api-key-cleanup-29724000-p7lq6" Jul 07 16:00:00.304520 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:00.304486 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9446\" (UniqueName: \"kubernetes.io/projected/7b16651f-b2cb-4e87-bc5a-3684c373e07a-kube-api-access-b9446\") pod \"maas-api-key-cleanup-29724000-p7lq6\" (UID: \"7b16651f-b2cb-4e87-bc5a-3684c373e07a\") " pod="opendatahub/maas-api-key-cleanup-29724000-p7lq6" Jul 07 16:00:00.313466 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:00.313441 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9446\" (UniqueName: \"kubernetes.io/projected/7b16651f-b2cb-4e87-bc5a-3684c373e07a-kube-api-access-b9446\") pod \"maas-api-key-cleanup-29724000-p7lq6\" (UID: \"7b16651f-b2cb-4e87-bc5a-3684c373e07a\") " pod="opendatahub/maas-api-key-cleanup-29724000-p7lq6" Jul 07 16:00:00.463293 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:00.463191 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724000-p7lq6" Jul 07 16:00:00.609952 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:00.609923 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724000-p7lq6"] Jul 07 16:00:00.611901 ip-10-0-139-248 kubenswrapper[2557]: W0707 16:00:00.611871 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b16651f_b2cb_4e87_bc5a_3684c373e07a.slice/crio-58b8724c28d1ea56cc8b5162afa9da863561f855a6e6438ad150dc4cd5b1332c WatchSource:0}: Error finding container 58b8724c28d1ea56cc8b5162afa9da863561f855a6e6438ad150dc4cd5b1332c: Status 404 returned error can't find the container with id 58b8724c28d1ea56cc8b5162afa9da863561f855a6e6438ad150dc4cd5b1332c Jul 07 16:00:00.613778 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:00.613760 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 16:00:01.185140 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:01.185091 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724000-p7lq6" event={"ID":"7b16651f-b2cb-4e87-bc5a-3684c373e07a","Type":"ContainerStarted","Data":"58b8724c28d1ea56cc8b5162afa9da863561f855a6e6438ad150dc4cd5b1332c"} Jul 07 16:00:12.251664 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:12.251618 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724000-p7lq6" event={"ID":"7b16651f-b2cb-4e87-bc5a-3684c373e07a","Type":"ContainerStarted","Data":"1dc35180691345136feaa03ac862da4d38398a1b3093f8e1e1ae251d3cd395ca"} Jul 07 16:00:12.267381 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:12.267330 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29724000-p7lq6" podStartSLOduration=1.40101377 podStartE2EDuration="12.267314885s" totalImagesPullingTime="10.866301115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:00:12.26690184 +0000 UTC m=+1196.444546326" watchObservedRunningTime="2026-07-07 16:00:12.267314885 +0000 UTC m=+1196.444959335" Jul 07 16:00:13.259525 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:13.259489 2557 generic.go:356] "Generic (PLEG): container finished" podID="7b16651f-b2cb-4e87-bc5a-3684c373e07a" containerID="1dc35180691345136feaa03ac862da4d38398a1b3093f8e1e1ae251d3cd395ca" exitCode=0 Jul 07 16:00:13.259951 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:13.259583 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724000-p7lq6" event={"ID":"7b16651f-b2cb-4e87-bc5a-3684c373e07a","Type":"ContainerDied","Data":"1dc35180691345136feaa03ac862da4d38398a1b3093f8e1e1ae251d3cd395ca"} Jul 07 16:00:14.410984 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:14.410957 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724000-p7lq6" Jul 07 16:00:14.537398 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:14.537362 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9446\" (UniqueName: \"kubernetes.io/projected/7b16651f-b2cb-4e87-bc5a-3684c373e07a-kube-api-access-b9446\") pod \"7b16651f-b2cb-4e87-bc5a-3684c373e07a\" (UID: \"7b16651f-b2cb-4e87-bc5a-3684c373e07a\") " Jul 07 16:00:14.539437 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:14.539413 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b16651f-b2cb-4e87-bc5a-3684c373e07a-kube-api-access-b9446" (OuterVolumeSpecName: "kube-api-access-b9446") pod "7b16651f-b2cb-4e87-bc5a-3684c373e07a" (UID: "7b16651f-b2cb-4e87-bc5a-3684c373e07a"). InnerVolumeSpecName "kube-api-access-b9446". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:00:14.638353 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:14.638318 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9446\" (UniqueName: \"kubernetes.io/projected/7b16651f-b2cb-4e87-bc5a-3684c373e07a-kube-api-access-b9446\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:00:15.272327 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:15.272292 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724000-p7lq6" Jul 07 16:00:15.272548 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:15.272326 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724000-p7lq6" event={"ID":"7b16651f-b2cb-4e87-bc5a-3684c373e07a","Type":"ContainerDied","Data":"58b8724c28d1ea56cc8b5162afa9da863561f855a6e6438ad150dc4cd5b1332c"} Jul 07 16:00:15.272548 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:15.272380 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58b8724c28d1ea56cc8b5162afa9da863561f855a6e6438ad150dc4cd5b1332c" Jul 07 16:00:16.378759 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:16.378732 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/1.log" Jul 07 16:00:16.384155 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:16.384107 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/1.log" Jul 07 16:00:16.387529 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:16.387505 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hkj2z_07a80436-65b9-4f0d-99b3-ecda36240b48/ovn-acl-logging/0.log" Jul 07 16:00:16.389499 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:00:16.389476 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hkj2z_07a80436-65b9-4f0d-99b3-ecda36240b48/ovn-acl-logging/0.log" Jul 07 16:05:05.546813 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:05.546776 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps"] Jul 07 16:05:05.547498 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:05.547471 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b16651f-b2cb-4e87-bc5a-3684c373e07a" containerName="cleanup" Jul 07 16:05:05.547594 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:05.547501 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b16651f-b2cb-4e87-bc5a-3684c373e07a" containerName="cleanup" Jul 07 16:05:05.547647 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:05.547634 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b16651f-b2cb-4e87-bc5a-3684c373e07a" containerName="cleanup" Jul 07 16:05:05.550355 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:05.550334 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps" Jul 07 16:05:05.553027 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:05.553004 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-7eb372bb-serving-cert\"" Jul 07 16:05:05.564380 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:05.564355 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps"] Jul 07 16:05:05.642268 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:05.642223 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8917e558-f54c-4365-9678-487ffab55527-maas-api-tls\") pod \"maas-api-e2e-ait-7eb372bb-569b47b785-2mnps\" (UID: \"8917e558-f54c-4365-9678-487ffab55527\") " pod="opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps" Jul 07 16:05:05.642446 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:05.642290 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l58v7\" (UniqueName: \"kubernetes.io/projected/8917e558-f54c-4365-9678-487ffab55527-kube-api-access-l58v7\") pod \"maas-api-e2e-ait-7eb372bb-569b47b785-2mnps\" (UID: \"8917e558-f54c-4365-9678-487ffab55527\") " pod="opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps" Jul 07 16:05:05.642446 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:05.642401 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8917e558-f54c-4365-9678-487ffab55527-openshift-service-ca\") pod \"maas-api-e2e-ait-7eb372bb-569b47b785-2mnps\" (UID: \"8917e558-f54c-4365-9678-487ffab55527\") " pod="opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps" Jul 07 16:05:05.743805 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:05.743760 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8917e558-f54c-4365-9678-487ffab55527-maas-api-tls\") pod \"maas-api-e2e-ait-7eb372bb-569b47b785-2mnps\" (UID: \"8917e558-f54c-4365-9678-487ffab55527\") " pod="opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps" Jul 07 16:05:05.744009 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:05.743831 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l58v7\" (UniqueName: \"kubernetes.io/projected/8917e558-f54c-4365-9678-487ffab55527-kube-api-access-l58v7\") pod \"maas-api-e2e-ait-7eb372bb-569b47b785-2mnps\" (UID: \"8917e558-f54c-4365-9678-487ffab55527\") " pod="opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps" Jul 07 16:05:05.744009 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:05.743886 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8917e558-f54c-4365-9678-487ffab55527-openshift-service-ca\") pod \"maas-api-e2e-ait-7eb372bb-569b47b785-2mnps\" (UID: \"8917e558-f54c-4365-9678-487ffab55527\") " pod="opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps" Jul 07 16:05:05.744009 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:05:05.743931 2557 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-7eb372bb-serving-cert: secret "maas-api-e2e-ait-7eb372bb-serving-cert" not found Jul 07 16:05:05.744191 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:05:05.744016 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8917e558-f54c-4365-9678-487ffab55527-maas-api-tls podName:8917e558-f54c-4365-9678-487ffab55527 nodeName:}" failed. No retries permitted until 2026-07-07 16:05:06.243994883 +0000 UTC m=+1490.421639313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8917e558-f54c-4365-9678-487ffab55527-maas-api-tls") pod "maas-api-e2e-ait-7eb372bb-569b47b785-2mnps" (UID: "8917e558-f54c-4365-9678-487ffab55527") : secret "maas-api-e2e-ait-7eb372bb-serving-cert" not found Jul 07 16:05:05.744807 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:05.744783 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8917e558-f54c-4365-9678-487ffab55527-openshift-service-ca\") pod \"maas-api-e2e-ait-7eb372bb-569b47b785-2mnps\" (UID: \"8917e558-f54c-4365-9678-487ffab55527\") " pod="opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps" Jul 07 16:05:05.752581 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:05.752553 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l58v7\" (UniqueName: \"kubernetes.io/projected/8917e558-f54c-4365-9678-487ffab55527-kube-api-access-l58v7\") pod \"maas-api-e2e-ait-7eb372bb-569b47b785-2mnps\" (UID: \"8917e558-f54c-4365-9678-487ffab55527\") " pod="opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps" Jul 07 16:05:06.248319 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:06.248276 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8917e558-f54c-4365-9678-487ffab55527-maas-api-tls\") pod \"maas-api-e2e-ait-7eb372bb-569b47b785-2mnps\" (UID: \"8917e558-f54c-4365-9678-487ffab55527\") " pod="opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps" Jul 07 16:05:06.250764 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:06.250741 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8917e558-f54c-4365-9678-487ffab55527-maas-api-tls\") pod \"maas-api-e2e-ait-7eb372bb-569b47b785-2mnps\" (UID: \"8917e558-f54c-4365-9678-487ffab55527\") " pod="opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps" Jul 07 16:05:06.461758 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:06.461722 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps" Jul 07 16:05:06.614466 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:06.614373 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps"] Jul 07 16:05:06.617200 ip-10-0-139-248 kubenswrapper[2557]: W0707 16:05:06.617166 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8917e558_f54c_4365_9678_487ffab55527.slice/crio-26a98c110f003b132bddb98b5a1e7133064f189a47db4d2d0f420f43c018395e WatchSource:0}: Error finding container 26a98c110f003b132bddb98b5a1e7133064f189a47db4d2d0f420f43c018395e: Status 404 returned error can't find the container with id 26a98c110f003b132bddb98b5a1e7133064f189a47db4d2d0f420f43c018395e Jul 07 16:05:06.618809 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:06.618777 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 16:05:07.004787 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:07.004755 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps" event={"ID":"8917e558-f54c-4365-9678-487ffab55527","Type":"ContainerStarted","Data":"26a98c110f003b132bddb98b5a1e7133064f189a47db4d2d0f420f43c018395e"} Jul 07 16:05:08.011735 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:08.011700 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps" event={"ID":"8917e558-f54c-4365-9678-487ffab55527","Type":"ContainerStarted","Data":"a525b21c1d2ff909cfb00c1baa2e2a5b5a7e2390b7d0a00ab37a2a9c6891c0ce"} Jul 07 16:05:08.012229 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:08.011778 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps" Jul 07 16:05:08.031574 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:08.031518 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps" podStartSLOduration=2.524242095 podStartE2EDuration="3.031501849s" totalImagesPullingTime="507.259754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:05:08.030054805 +0000 UTC m=+1492.207699258" watchObservedRunningTime="2026-07-07 16:05:08.031501849 +0000 UTC m=+1492.209146298" Jul 07 16:05:10.916411 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:10.916372 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps"] Jul 07 16:05:10.916988 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:10.916601 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps" podUID="8917e558-f54c-4365-9678-487ffab55527" containerName="maas-api" containerID="cri-o://a525b21c1d2ff909cfb00c1baa2e2a5b5a7e2390b7d0a00ab37a2a9c6891c0ce" gracePeriod=30 Jul 07 16:05:11.165933 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:11.165906 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps" Jul 07 16:05:11.297329 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:11.297300 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8917e558-f54c-4365-9678-487ffab55527-openshift-service-ca\") pod \"8917e558-f54c-4365-9678-487ffab55527\" (UID: \"8917e558-f54c-4365-9678-487ffab55527\") " Jul 07 16:05:11.297501 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:11.297344 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l58v7\" (UniqueName: \"kubernetes.io/projected/8917e558-f54c-4365-9678-487ffab55527-kube-api-access-l58v7\") pod \"8917e558-f54c-4365-9678-487ffab55527\" (UID: \"8917e558-f54c-4365-9678-487ffab55527\") " Jul 07 16:05:11.297501 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:11.297396 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8917e558-f54c-4365-9678-487ffab55527-maas-api-tls\") pod \"8917e558-f54c-4365-9678-487ffab55527\" (UID: \"8917e558-f54c-4365-9678-487ffab55527\") " Jul 07 16:05:11.297679 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:11.297653 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8917e558-f54c-4365-9678-487ffab55527-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8917e558-f54c-4365-9678-487ffab55527" (UID: "8917e558-f54c-4365-9678-487ffab55527"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:05:11.297773 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:11.297750 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8917e558-f54c-4365-9678-487ffab55527-openshift-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:05:11.299557 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:11.299527 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8917e558-f54c-4365-9678-487ffab55527-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8917e558-f54c-4365-9678-487ffab55527" (UID: "8917e558-f54c-4365-9678-487ffab55527"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:05:11.299643 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:11.299563 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8917e558-f54c-4365-9678-487ffab55527-kube-api-access-l58v7" (OuterVolumeSpecName: "kube-api-access-l58v7") pod "8917e558-f54c-4365-9678-487ffab55527" (UID: "8917e558-f54c-4365-9678-487ffab55527"). InnerVolumeSpecName "kube-api-access-l58v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:05:11.399165 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:11.399095 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8917e558-f54c-4365-9678-487ffab55527-maas-api-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:05:11.399165 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:11.399171 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l58v7\" (UniqueName: \"kubernetes.io/projected/8917e558-f54c-4365-9678-487ffab55527-kube-api-access-l58v7\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:05:12.035808 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:12.035772 2557 generic.go:356] "Generic (PLEG): container finished" podID="8917e558-f54c-4365-9678-487ffab55527" containerID="a525b21c1d2ff909cfb00c1baa2e2a5b5a7e2390b7d0a00ab37a2a9c6891c0ce" exitCode=0 Jul 07 16:05:12.036262 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:12.035820 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps" event={"ID":"8917e558-f54c-4365-9678-487ffab55527","Type":"ContainerDied","Data":"a525b21c1d2ff909cfb00c1baa2e2a5b5a7e2390b7d0a00ab37a2a9c6891c0ce"} Jul 07 16:05:12.036262 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:12.035822 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps" Jul 07 16:05:12.036262 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:12.035863 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps" event={"ID":"8917e558-f54c-4365-9678-487ffab55527","Type":"ContainerDied","Data":"26a98c110f003b132bddb98b5a1e7133064f189a47db4d2d0f420f43c018395e"} Jul 07 16:05:12.036262 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:12.035886 2557 scope.go:117] "RemoveContainer" containerID="a525b21c1d2ff909cfb00c1baa2e2a5b5a7e2390b7d0a00ab37a2a9c6891c0ce" Jul 07 16:05:12.046395 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:12.046375 2557 scope.go:117] "RemoveContainer" containerID="a525b21c1d2ff909cfb00c1baa2e2a5b5a7e2390b7d0a00ab37a2a9c6891c0ce" Jul 07 16:05:12.046641 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:05:12.046622 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a525b21c1d2ff909cfb00c1baa2e2a5b5a7e2390b7d0a00ab37a2a9c6891c0ce\": container with ID starting with a525b21c1d2ff909cfb00c1baa2e2a5b5a7e2390b7d0a00ab37a2a9c6891c0ce not found: ID does not exist" containerID="a525b21c1d2ff909cfb00c1baa2e2a5b5a7e2390b7d0a00ab37a2a9c6891c0ce" Jul 07 16:05:12.046688 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:12.046650 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a525b21c1d2ff909cfb00c1baa2e2a5b5a7e2390b7d0a00ab37a2a9c6891c0ce"} err="failed to get container status \"a525b21c1d2ff909cfb00c1baa2e2a5b5a7e2390b7d0a00ab37a2a9c6891c0ce\": rpc error: code = NotFound desc = could not find container \"a525b21c1d2ff909cfb00c1baa2e2a5b5a7e2390b7d0a00ab37a2a9c6891c0ce\": container with ID starting with a525b21c1d2ff909cfb00c1baa2e2a5b5a7e2390b7d0a00ab37a2a9c6891c0ce not found: ID does not exist" Jul 07 16:05:12.057987 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:12.057964 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps"] Jul 07 16:05:12.061682 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:12.061659 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-7eb372bb-569b47b785-2mnps"] Jul 07 16:05:12.351303 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:12.351224 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8917e558-f54c-4365-9678-487ffab55527" path="/var/lib/kubelet/pods/8917e558-f54c-4365-9678-487ffab55527/volumes" Jul 07 16:05:16.420104 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:16.420077 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/1.log" Jul 07 16:05:16.423564 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:16.423539 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/1.log" Jul 07 16:05:16.425094 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:16.425072 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hkj2z_07a80436-65b9-4f0d-99b3-ecda36240b48/ovn-acl-logging/0.log" Jul 07 16:05:16.428571 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:16.428542 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hkj2z_07a80436-65b9-4f0d-99b3-ecda36240b48/ovn-acl-logging/0.log" Jul 07 16:05:18.260134 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:18.260072 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8"] Jul 07 16:05:18.260766 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:18.260744 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8917e558-f54c-4365-9678-487ffab55527" containerName="maas-api" Jul 07 16:05:18.260766 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:18.260767 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8917e558-f54c-4365-9678-487ffab55527" containerName="maas-api" Jul 07 16:05:18.260901 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:18.260869 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8917e558-f54c-4365-9678-487ffab55527" containerName="maas-api" Jul 07 16:05:18.264379 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:18.264356 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8" Jul 07 16:05:18.266355 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:18.266331 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-18541f86-serving-cert\"" Jul 07 16:05:18.276050 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:18.275480 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8"] Jul 07 16:05:18.365384 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:18.365349 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8275a55-cfc6-47b7-ad13-73850f818a20-openshift-service-ca\") pod \"maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8\" (UID: \"c8275a55-cfc6-47b7-ad13-73850f818a20\") " pod="opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8" Jul 07 16:05:18.365669 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:18.365641 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdw48\" (UniqueName: \"kubernetes.io/projected/c8275a55-cfc6-47b7-ad13-73850f818a20-kube-api-access-tdw48\") pod \"maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8\" (UID: \"c8275a55-cfc6-47b7-ad13-73850f818a20\") " pod="opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8" Jul 07 16:05:18.365888 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:18.365773 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8275a55-cfc6-47b7-ad13-73850f818a20-maas-api-tls\") pod \"maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8\" (UID: \"c8275a55-cfc6-47b7-ad13-73850f818a20\") " pod="opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8" Jul 07 16:05:18.466757 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:18.466727 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8275a55-cfc6-47b7-ad13-73850f818a20-maas-api-tls\") pod \"maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8\" (UID: \"c8275a55-cfc6-47b7-ad13-73850f818a20\") " pod="opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8" Jul 07 16:05:18.466947 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:18.466776 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8275a55-cfc6-47b7-ad13-73850f818a20-openshift-service-ca\") pod \"maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8\" (UID: \"c8275a55-cfc6-47b7-ad13-73850f818a20\") " pod="opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8" Jul 07 16:05:18.466947 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:18.466807 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdw48\" (UniqueName: \"kubernetes.io/projected/c8275a55-cfc6-47b7-ad13-73850f818a20-kube-api-access-tdw48\") pod \"maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8\" (UID: \"c8275a55-cfc6-47b7-ad13-73850f818a20\") " pod="opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8" Jul 07 16:05:18.466947 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:05:18.466912 2557 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-18541f86-serving-cert: secret "maas-api-e2e-ait-18541f86-serving-cert" not found Jul 07 16:05:18.467173 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:05:18.466988 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8275a55-cfc6-47b7-ad13-73850f818a20-maas-api-tls podName:c8275a55-cfc6-47b7-ad13-73850f818a20 nodeName:}" failed. No retries permitted until 2026-07-07 16:05:18.96696819 +0000 UTC m=+1503.144612623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8275a55-cfc6-47b7-ad13-73850f818a20-maas-api-tls") pod "maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8" (UID: "c8275a55-cfc6-47b7-ad13-73850f818a20") : secret "maas-api-e2e-ait-18541f86-serving-cert" not found Jul 07 16:05:18.467685 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:18.467661 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8275a55-cfc6-47b7-ad13-73850f818a20-openshift-service-ca\") pod \"maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8\" (UID: \"c8275a55-cfc6-47b7-ad13-73850f818a20\") " pod="opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8" Jul 07 16:05:18.477848 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:18.477817 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdw48\" (UniqueName: \"kubernetes.io/projected/c8275a55-cfc6-47b7-ad13-73850f818a20-kube-api-access-tdw48\") pod \"maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8\" (UID: \"c8275a55-cfc6-47b7-ad13-73850f818a20\") " pod="opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8" Jul 07 16:05:18.972451 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:18.972417 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8275a55-cfc6-47b7-ad13-73850f818a20-maas-api-tls\") pod \"maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8\" (UID: \"c8275a55-cfc6-47b7-ad13-73850f818a20\") " pod="opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8" Jul 07 16:05:18.975321 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:18.975293 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8275a55-cfc6-47b7-ad13-73850f818a20-maas-api-tls\") pod \"maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8\" (UID: \"c8275a55-cfc6-47b7-ad13-73850f818a20\") " pod="opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8" Jul 07 16:05:19.183178 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:19.183135 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8" Jul 07 16:05:19.336247 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:19.336185 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8"] Jul 07 16:05:19.337571 ip-10-0-139-248 kubenswrapper[2557]: W0707 16:05:19.337539 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8275a55_cfc6_47b7_ad13_73850f818a20.slice/crio-1c96d7655a34c8d4086b890ee1160bcb6bffb916f1b233f385b4fcbb9c0c9f07 WatchSource:0}: Error finding container 1c96d7655a34c8d4086b890ee1160bcb6bffb916f1b233f385b4fcbb9c0c9f07: Status 404 returned error can't find the container with id 1c96d7655a34c8d4086b890ee1160bcb6bffb916f1b233f385b4fcbb9c0c9f07 Jul 07 16:05:20.086984 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:20.086950 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8" event={"ID":"c8275a55-cfc6-47b7-ad13-73850f818a20","Type":"ContainerStarted","Data":"2dd1c0c878c617f7a9ad488b25d5b0bb3372e7c7682e88285d54ed63eb4116e2"} Jul 07 16:05:20.087213 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:20.087006 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8" event={"ID":"c8275a55-cfc6-47b7-ad13-73850f818a20","Type":"ContainerStarted","Data":"1c96d7655a34c8d4086b890ee1160bcb6bffb916f1b233f385b4fcbb9c0c9f07"} Jul 07 16:05:20.087213 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:20.087031 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8" Jul 07 16:05:20.127977 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:20.127921 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8" podStartSLOduration=1.767228391 podStartE2EDuration="2.127879609s" totalImagesPullingTime="360.651218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:05:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:05:20.125219284 +0000 UTC m=+1504.302863733" watchObservedRunningTime="2026-07-07 16:05:20.127879609 +0000 UTC m=+1504.305524058" Jul 07 16:05:23.602739 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:23.602700 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8"] Jul 07 16:05:23.603302 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:23.602985 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8" podUID="c8275a55-cfc6-47b7-ad13-73850f818a20" containerName="maas-api" containerID="cri-o://2dd1c0c878c617f7a9ad488b25d5b0bb3372e7c7682e88285d54ed63eb4116e2" gracePeriod=30 Jul 07 16:05:23.841888 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:23.841864 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8" Jul 07 16:05:24.023738 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:24.023641 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8275a55-cfc6-47b7-ad13-73850f818a20-maas-api-tls\") pod \"c8275a55-cfc6-47b7-ad13-73850f818a20\" (UID: \"c8275a55-cfc6-47b7-ad13-73850f818a20\") " Jul 07 16:05:24.023738 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:24.023692 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdw48\" (UniqueName: \"kubernetes.io/projected/c8275a55-cfc6-47b7-ad13-73850f818a20-kube-api-access-tdw48\") pod \"c8275a55-cfc6-47b7-ad13-73850f818a20\" (UID: \"c8275a55-cfc6-47b7-ad13-73850f818a20\") " Jul 07 16:05:24.023994 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:24.023747 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8275a55-cfc6-47b7-ad13-73850f818a20-openshift-service-ca\") pod \"c8275a55-cfc6-47b7-ad13-73850f818a20\" (UID: \"c8275a55-cfc6-47b7-ad13-73850f818a20\") " Jul 07 16:05:24.024146 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:24.024092 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8275a55-cfc6-47b7-ad13-73850f818a20-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c8275a55-cfc6-47b7-ad13-73850f818a20" (UID: "c8275a55-cfc6-47b7-ad13-73850f818a20"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:05:24.025966 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:24.025938 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8275a55-cfc6-47b7-ad13-73850f818a20-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c8275a55-cfc6-47b7-ad13-73850f818a20" (UID: "c8275a55-cfc6-47b7-ad13-73850f818a20"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:05:24.025966 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:24.025960 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8275a55-cfc6-47b7-ad13-73850f818a20-kube-api-access-tdw48" (OuterVolumeSpecName: "kube-api-access-tdw48") pod "c8275a55-cfc6-47b7-ad13-73850f818a20" (UID: "c8275a55-cfc6-47b7-ad13-73850f818a20"). InnerVolumeSpecName "kube-api-access-tdw48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:05:24.113894 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:24.113859 2557 generic.go:356] "Generic (PLEG): container finished" podID="c8275a55-cfc6-47b7-ad13-73850f818a20" containerID="2dd1c0c878c617f7a9ad488b25d5b0bb3372e7c7682e88285d54ed63eb4116e2" exitCode=0 Jul 07 16:05:24.114067 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:24.113899 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8" Jul 07 16:05:24.114067 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:24.113942 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8" event={"ID":"c8275a55-cfc6-47b7-ad13-73850f818a20","Type":"ContainerDied","Data":"2dd1c0c878c617f7a9ad488b25d5b0bb3372e7c7682e88285d54ed63eb4116e2"} Jul 07 16:05:24.114067 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:24.113995 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8" event={"ID":"c8275a55-cfc6-47b7-ad13-73850f818a20","Type":"ContainerDied","Data":"1c96d7655a34c8d4086b890ee1160bcb6bffb916f1b233f385b4fcbb9c0c9f07"} Jul 07 16:05:24.114067 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:24.114019 2557 scope.go:117] "RemoveContainer" containerID="2dd1c0c878c617f7a9ad488b25d5b0bb3372e7c7682e88285d54ed63eb4116e2" Jul 07 16:05:24.123896 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:24.123879 2557 scope.go:117] "RemoveContainer" containerID="2dd1c0c878c617f7a9ad488b25d5b0bb3372e7c7682e88285d54ed63eb4116e2" Jul 07 16:05:24.124169 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:05:24.124150 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd1c0c878c617f7a9ad488b25d5b0bb3372e7c7682e88285d54ed63eb4116e2\": container with ID starting with 2dd1c0c878c617f7a9ad488b25d5b0bb3372e7c7682e88285d54ed63eb4116e2 not found: ID does not exist" containerID="2dd1c0c878c617f7a9ad488b25d5b0bb3372e7c7682e88285d54ed63eb4116e2" Jul 07 16:05:24.124241 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:24.124178 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd1c0c878c617f7a9ad488b25d5b0bb3372e7c7682e88285d54ed63eb4116e2"} err="failed to get container status \"2dd1c0c878c617f7a9ad488b25d5b0bb3372e7c7682e88285d54ed63eb4116e2\": rpc error: code = NotFound desc = could not find container \"2dd1c0c878c617f7a9ad488b25d5b0bb3372e7c7682e88285d54ed63eb4116e2\": container with ID starting with 2dd1c0c878c617f7a9ad488b25d5b0bb3372e7c7682e88285d54ed63eb4116e2 not found: ID does not exist" Jul 07 16:05:24.125699 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:24.125680 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8275a55-cfc6-47b7-ad13-73850f818a20-maas-api-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:05:24.125699 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:24.125701 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdw48\" (UniqueName: \"kubernetes.io/projected/c8275a55-cfc6-47b7-ad13-73850f818a20-kube-api-access-tdw48\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:05:24.125838 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:24.125710 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8275a55-cfc6-47b7-ad13-73850f818a20-openshift-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:05:24.135788 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:24.135764 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8"] Jul 07 16:05:24.140203 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:24.140180 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-18541f86-5dc8f9c9c6-7x5m8"] Jul 07 16:05:24.352081 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:24.352052 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8275a55-cfc6-47b7-ad13-73850f818a20" path="/var/lib/kubelet/pods/c8275a55-cfc6-47b7-ad13-73850f818a20/volumes" Jul 07 16:05:36.846479 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:36.846446 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r"] Jul 07 16:05:36.846957 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:36.846866 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8275a55-cfc6-47b7-ad13-73850f818a20" containerName="maas-api" Jul 07 16:05:36.846957 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:36.846878 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8275a55-cfc6-47b7-ad13-73850f818a20" containerName="maas-api" Jul 07 16:05:36.846957 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:36.846958 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8275a55-cfc6-47b7-ad13-73850f818a20" containerName="maas-api" Jul 07 16:05:36.851435 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:36.851417 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r" Jul 07 16:05:36.853747 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:36.853725 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-15eb70f8-serving-cert\"" Jul 07 16:05:36.874431 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:36.874406 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r"] Jul 07 16:05:36.931888 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:36.931855 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l982\" (UniqueName: \"kubernetes.io/projected/be8ac256-14e2-4c07-8915-e055b048f4f9-kube-api-access-7l982\") pod \"maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r\" (UID: \"be8ac256-14e2-4c07-8915-e055b048f4f9\") " pod="opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r" Jul 07 16:05:36.932088 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:36.931920 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8ac256-14e2-4c07-8915-e055b048f4f9-openshift-service-ca\") pod \"maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r\" (UID: \"be8ac256-14e2-4c07-8915-e055b048f4f9\") " pod="opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r" Jul 07 16:05:36.932088 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:36.931969 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be8ac256-14e2-4c07-8915-e055b048f4f9-maas-api-tls\") pod \"maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r\" (UID: \"be8ac256-14e2-4c07-8915-e055b048f4f9\") " pod="opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r" Jul 07 16:05:37.032504 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:37.032470 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l982\" (UniqueName: \"kubernetes.io/projected/be8ac256-14e2-4c07-8915-e055b048f4f9-kube-api-access-7l982\") pod \"maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r\" (UID: \"be8ac256-14e2-4c07-8915-e055b048f4f9\") " pod="opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r" Jul 07 16:05:37.032692 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:37.032540 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8ac256-14e2-4c07-8915-e055b048f4f9-openshift-service-ca\") pod \"maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r\" (UID: \"be8ac256-14e2-4c07-8915-e055b048f4f9\") " pod="opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r" Jul 07 16:05:37.032692 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:37.032572 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be8ac256-14e2-4c07-8915-e055b048f4f9-maas-api-tls\") pod \"maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r\" (UID: \"be8ac256-14e2-4c07-8915-e055b048f4f9\") " pod="opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r" Jul 07 16:05:37.032801 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:05:37.032771 2557 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-15eb70f8-serving-cert: secret "maas-api-e2e-derive-15eb70f8-serving-cert" not found Jul 07 16:05:37.032899 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:05:37.032885 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be8ac256-14e2-4c07-8915-e055b048f4f9-maas-api-tls podName:be8ac256-14e2-4c07-8915-e055b048f4f9 nodeName:}" failed. No retries permitted until 2026-07-07 16:05:37.532864431 +0000 UTC m=+1521.710508859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/be8ac256-14e2-4c07-8915-e055b048f4f9-maas-api-tls") pod "maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r" (UID: "be8ac256-14e2-4c07-8915-e055b048f4f9") : secret "maas-api-e2e-derive-15eb70f8-serving-cert" not found Jul 07 16:05:37.033340 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:37.033323 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8ac256-14e2-4c07-8915-e055b048f4f9-openshift-service-ca\") pod \"maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r\" (UID: \"be8ac256-14e2-4c07-8915-e055b048f4f9\") " pod="opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r" Jul 07 16:05:37.046065 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:37.046037 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l982\" (UniqueName: \"kubernetes.io/projected/be8ac256-14e2-4c07-8915-e055b048f4f9-kube-api-access-7l982\") pod \"maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r\" (UID: \"be8ac256-14e2-4c07-8915-e055b048f4f9\") " pod="opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r" Jul 07 16:05:37.538394 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:37.538356 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be8ac256-14e2-4c07-8915-e055b048f4f9-maas-api-tls\") pod \"maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r\" (UID: \"be8ac256-14e2-4c07-8915-e055b048f4f9\") " pod="opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r" Jul 07 16:05:37.540901 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:37.540865 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be8ac256-14e2-4c07-8915-e055b048f4f9-maas-api-tls\") pod \"maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r\" (UID: \"be8ac256-14e2-4c07-8915-e055b048f4f9\") " pod="opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r" Jul 07 16:05:37.762235 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:37.762201 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r" Jul 07 16:05:38.121493 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:38.121468 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r"] Jul 07 16:05:38.124231 ip-10-0-139-248 kubenswrapper[2557]: W0707 16:05:38.124175 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8ac256_14e2_4c07_8915_e055b048f4f9.slice/crio-801f73961286257fa1b3260ef7def758b387c6bdc38a03b9c139b989cb9e203f WatchSource:0}: Error finding container 801f73961286257fa1b3260ef7def758b387c6bdc38a03b9c139b989cb9e203f: Status 404 returned error can't find the container with id 801f73961286257fa1b3260ef7def758b387c6bdc38a03b9c139b989cb9e203f Jul 07 16:05:38.200070 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:38.199990 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r" event={"ID":"be8ac256-14e2-4c07-8915-e055b048f4f9","Type":"ContainerStarted","Data":"801f73961286257fa1b3260ef7def758b387c6bdc38a03b9c139b989cb9e203f"} Jul 07 16:05:39.213798 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:39.213700 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r" event={"ID":"be8ac256-14e2-4c07-8915-e055b048f4f9","Type":"ContainerStarted","Data":"4a9c315adc4fc990eadef48f27e2f5bbc28210b8ecadaa3f90ef3b16b603a9bd"} Jul 07 16:05:39.214188 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:39.213810 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r" Jul 07 16:05:39.232018 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:39.231967 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r" podStartSLOduration=2.96010959 podStartE2EDuration="3.231951013s" totalImagesPullingTime="271.841423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:05:39.230639795 +0000 UTC m=+1523.408284245" watchObservedRunningTime="2026-07-07 16:05:39.231951013 +0000 UTC m=+1523.409595463" Jul 07 16:05:41.555522 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:41.555483 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r"] Jul 07 16:05:41.556029 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:41.555702 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r" podUID="be8ac256-14e2-4c07-8915-e055b048f4f9" containerName="maas-api" containerID="cri-o://4a9c315adc4fc990eadef48f27e2f5bbc28210b8ecadaa3f90ef3b16b603a9bd" gracePeriod=30 Jul 07 16:05:41.815291 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:41.815232 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r" Jul 07 16:05:41.873880 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:41.873847 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8ac256-14e2-4c07-8915-e055b048f4f9-openshift-service-ca\") pod \"be8ac256-14e2-4c07-8915-e055b048f4f9\" (UID: \"be8ac256-14e2-4c07-8915-e055b048f4f9\") " Jul 07 16:05:41.874077 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:41.873946 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l982\" (UniqueName: \"kubernetes.io/projected/be8ac256-14e2-4c07-8915-e055b048f4f9-kube-api-access-7l982\") pod \"be8ac256-14e2-4c07-8915-e055b048f4f9\" (UID: \"be8ac256-14e2-4c07-8915-e055b048f4f9\") " Jul 07 16:05:41.874077 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:41.873993 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be8ac256-14e2-4c07-8915-e055b048f4f9-maas-api-tls\") pod \"be8ac256-14e2-4c07-8915-e055b048f4f9\" (UID: \"be8ac256-14e2-4c07-8915-e055b048f4f9\") " Jul 07 16:05:41.874295 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:41.874271 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8ac256-14e2-4c07-8915-e055b048f4f9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "be8ac256-14e2-4c07-8915-e055b048f4f9" (UID: "be8ac256-14e2-4c07-8915-e055b048f4f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:05:41.876203 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:41.876177 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8ac256-14e2-4c07-8915-e055b048f4f9-kube-api-access-7l982" (OuterVolumeSpecName: "kube-api-access-7l982") pod "be8ac256-14e2-4c07-8915-e055b048f4f9" (UID: "be8ac256-14e2-4c07-8915-e055b048f4f9"). InnerVolumeSpecName "kube-api-access-7l982". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:05:41.876302 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:41.876202 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8ac256-14e2-4c07-8915-e055b048f4f9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "be8ac256-14e2-4c07-8915-e055b048f4f9" (UID: "be8ac256-14e2-4c07-8915-e055b048f4f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:05:41.975522 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:41.975487 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l982\" (UniqueName: \"kubernetes.io/projected/be8ac256-14e2-4c07-8915-e055b048f4f9-kube-api-access-7l982\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:05:41.975522 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:41.975516 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be8ac256-14e2-4c07-8915-e055b048f4f9-maas-api-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:05:41.975522 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:41.975525 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8ac256-14e2-4c07-8915-e055b048f4f9-openshift-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:05:42.232906 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:42.232813 2557 generic.go:356] "Generic (PLEG): container finished" podID="be8ac256-14e2-4c07-8915-e055b048f4f9" containerID="4a9c315adc4fc990eadef48f27e2f5bbc28210b8ecadaa3f90ef3b16b603a9bd" exitCode=0 Jul 07 16:05:42.232906 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:42.232861 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r" Jul 07 16:05:42.233146 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:42.232901 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r" event={"ID":"be8ac256-14e2-4c07-8915-e055b048f4f9","Type":"ContainerDied","Data":"4a9c315adc4fc990eadef48f27e2f5bbc28210b8ecadaa3f90ef3b16b603a9bd"} Jul 07 16:05:42.233146 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:42.232957 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r" event={"ID":"be8ac256-14e2-4c07-8915-e055b048f4f9","Type":"ContainerDied","Data":"801f73961286257fa1b3260ef7def758b387c6bdc38a03b9c139b989cb9e203f"} Jul 07 16:05:42.233146 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:42.232984 2557 scope.go:117] "RemoveContainer" containerID="4a9c315adc4fc990eadef48f27e2f5bbc28210b8ecadaa3f90ef3b16b603a9bd" Jul 07 16:05:42.242893 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:42.242873 2557 scope.go:117] "RemoveContainer" containerID="4a9c315adc4fc990eadef48f27e2f5bbc28210b8ecadaa3f90ef3b16b603a9bd" Jul 07 16:05:42.243172 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:05:42.243143 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a9c315adc4fc990eadef48f27e2f5bbc28210b8ecadaa3f90ef3b16b603a9bd\": container with ID starting with 4a9c315adc4fc990eadef48f27e2f5bbc28210b8ecadaa3f90ef3b16b603a9bd not found: ID does not exist" containerID="4a9c315adc4fc990eadef48f27e2f5bbc28210b8ecadaa3f90ef3b16b603a9bd" Jul 07 16:05:42.243246 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:42.243172 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a9c315adc4fc990eadef48f27e2f5bbc28210b8ecadaa3f90ef3b16b603a9bd"} err="failed to get container status \"4a9c315adc4fc990eadef48f27e2f5bbc28210b8ecadaa3f90ef3b16b603a9bd\": rpc error: code = NotFound desc = could not find container \"4a9c315adc4fc990eadef48f27e2f5bbc28210b8ecadaa3f90ef3b16b603a9bd\": container with ID starting with 4a9c315adc4fc990eadef48f27e2f5bbc28210b8ecadaa3f90ef3b16b603a9bd not found: ID does not exist" Jul 07 16:05:42.254742 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:42.254720 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r"] Jul 07 16:05:42.257846 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:42.257826 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-15eb70f8-5769744d58-hfb7r"] Jul 07 16:05:42.351763 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:05:42.351728 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be8ac256-14e2-4c07-8915-e055b048f4f9" path="/var/lib/kubelet/pods/be8ac256-14e2-4c07-8915-e055b048f4f9/volumes" Jul 07 16:07:27.396977 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:27.396896 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8"] Jul 07 16:07:27.397465 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:27.397404 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be8ac256-14e2-4c07-8915-e055b048f4f9" containerName="maas-api" Jul 07 16:07:27.397465 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:27.397419 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="be8ac256-14e2-4c07-8915-e055b048f4f9" containerName="maas-api" Jul 07 16:07:27.397546 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:27.397501 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="be8ac256-14e2-4c07-8915-e055b048f4f9" containerName="maas-api" Jul 07 16:07:27.400679 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:27.400657 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" Jul 07 16:07:27.403642 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:27.403621 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-1cc03dc5-serving-cert\"" Jul 07 16:07:27.411283 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:27.411254 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8"] Jul 07 16:07:27.565867 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:27.565828 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3-openshift-service-ca\") pod \"maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8\" (UID: \"7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3\") " pod="opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" Jul 07 16:07:27.566077 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:27.565960 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3-maas-api-tls\") pod \"maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8\" (UID: \"7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3\") " pod="opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" Jul 07 16:07:27.566077 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:27.566024 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zpk4\" (UniqueName: \"kubernetes.io/projected/7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3-kube-api-access-2zpk4\") pod \"maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8\" (UID: \"7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3\") " pod="opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" Jul 07 16:07:27.667291 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:27.667199 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3-maas-api-tls\") pod \"maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8\" (UID: \"7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3\") " pod="opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" Jul 07 16:07:27.667291 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:27.667243 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zpk4\" (UniqueName: \"kubernetes.io/projected/7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3-kube-api-access-2zpk4\") pod \"maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8\" (UID: \"7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3\") " pod="opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" Jul 07 16:07:27.667498 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:27.667323 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3-openshift-service-ca\") pod \"maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8\" (UID: \"7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3\") " pod="opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" Jul 07 16:07:27.667498 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:07:27.667353 2557 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-1cc03dc5-serving-cert: secret "maas-api-e2e-mt-1cc03dc5-serving-cert" not found Jul 07 16:07:27.667498 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:07:27.667425 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3-maas-api-tls podName:7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3 nodeName:}" failed. No retries permitted until 2026-07-07 16:07:28.167408275 +0000 UTC m=+1632.345052703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3-maas-api-tls") pod "maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" (UID: "7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3") : secret "maas-api-e2e-mt-1cc03dc5-serving-cert" not found Jul 07 16:07:27.668013 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:27.667995 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3-openshift-service-ca\") pod \"maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8\" (UID: \"7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3\") " pod="opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" Jul 07 16:07:27.684671 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:27.684643 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zpk4\" (UniqueName: \"kubernetes.io/projected/7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3-kube-api-access-2zpk4\") pod \"maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8\" (UID: \"7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3\") " pod="opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" Jul 07 16:07:28.170533 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:28.170500 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3-maas-api-tls\") pod \"maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8\" (UID: \"7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3\") " pod="opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" Jul 07 16:07:28.173270 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:28.173238 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3-maas-api-tls\") pod \"maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8\" (UID: \"7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3\") " pod="opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" Jul 07 16:07:28.312704 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:28.312668 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" Jul 07 16:07:28.471060 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:28.471026 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8"] Jul 07 16:07:28.472367 ip-10-0-139-248 kubenswrapper[2557]: W0707 16:07:28.472330 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a93680b_3f9b_4ba8_9fd9_dfe11aafcbe3.slice/crio-8c373b85b3dc398f162a9b70d77ef9692b252ddde9ae7e20a94bcca726cc7ba7 WatchSource:0}: Error finding container 8c373b85b3dc398f162a9b70d77ef9692b252ddde9ae7e20a94bcca726cc7ba7: Status 404 returned error can't find the container with id 8c373b85b3dc398f162a9b70d77ef9692b252ddde9ae7e20a94bcca726cc7ba7 Jul 07 16:07:28.845716 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:28.845678 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" event={"ID":"7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3","Type":"ContainerStarted","Data":"8c373b85b3dc398f162a9b70d77ef9692b252ddde9ae7e20a94bcca726cc7ba7"} Jul 07 16:07:29.852869 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:29.852825 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" event={"ID":"7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3","Type":"ContainerStarted","Data":"61e311dec2dac31a9b309572366e6be041d9babfc30bebe1d6d5a32070feedb0"} Jul 07 16:07:29.853277 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:29.852896 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" Jul 07 16:07:29.872674 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:29.872625 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" podStartSLOduration=2.4354856639999998 podStartE2EDuration="2.872608493s" totalImagesPullingTime="437.122829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:07:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:07:29.871776272 +0000 UTC m=+1634.049420721" watchObservedRunningTime="2026-07-07 16:07:29.872608493 +0000 UTC m=+1634.050252943" Jul 07 16:07:35.863603 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:35.863576 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" Jul 07 16:07:39.686467 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:39.686428 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8"] Jul 07 16:07:39.686974 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:39.686726 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" podUID="7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3" containerName="maas-api" containerID="cri-o://61e311dec2dac31a9b309572366e6be041d9babfc30bebe1d6d5a32070feedb0" gracePeriod=30 Jul 07 16:07:39.917778 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:39.917743 2557 generic.go:356] "Generic (PLEG): container finished" podID="7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3" containerID="61e311dec2dac31a9b309572366e6be041d9babfc30bebe1d6d5a32070feedb0" exitCode=0 Jul 07 16:07:39.917976 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:39.917815 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" event={"ID":"7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3","Type":"ContainerDied","Data":"61e311dec2dac31a9b309572366e6be041d9babfc30bebe1d6d5a32070feedb0"} Jul 07 16:07:39.945909 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:39.945844 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" Jul 07 16:07:40.085232 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:40.085201 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3-openshift-service-ca\") pod \"7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3\" (UID: \"7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3\") " Jul 07 16:07:40.085410 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:40.085278 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zpk4\" (UniqueName: \"kubernetes.io/projected/7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3-kube-api-access-2zpk4\") pod \"7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3\" (UID: \"7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3\") " Jul 07 16:07:40.085410 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:40.085353 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3-maas-api-tls\") pod \"7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3\" (UID: \"7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3\") " Jul 07 16:07:40.085628 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:40.085601 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3" (UID: "7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:07:40.085704 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:40.085689 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3-openshift-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:07:40.087482 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:40.087448 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3" (UID: "7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:07:40.087595 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:40.087488 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3-kube-api-access-2zpk4" (OuterVolumeSpecName: "kube-api-access-2zpk4") pod "7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3" (UID: "7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3"). InnerVolumeSpecName "kube-api-access-2zpk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:07:40.187063 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:40.187027 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zpk4\" (UniqueName: \"kubernetes.io/projected/7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3-kube-api-access-2zpk4\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:07:40.187063 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:40.187061 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3-maas-api-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:07:40.924882 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:40.924853 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" Jul 07 16:07:40.924882 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:40.924872 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8" event={"ID":"7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3","Type":"ContainerDied","Data":"8c373b85b3dc398f162a9b70d77ef9692b252ddde9ae7e20a94bcca726cc7ba7"} Jul 07 16:07:40.925415 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:40.924925 2557 scope.go:117] "RemoveContainer" containerID="61e311dec2dac31a9b309572366e6be041d9babfc30bebe1d6d5a32070feedb0" Jul 07 16:07:40.942275 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:40.942253 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8"] Jul 07 16:07:40.949250 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:40.949231 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-1cc03dc5-5979cbc64d-s5vc8"] Jul 07 16:07:42.350931 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:07:42.350895 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3" path="/var/lib/kubelet/pods/7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3/volumes" Jul 07 16:08:08.577050 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:08.577007 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54"] Jul 07 16:08:08.577540 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:08.577458 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3" containerName="maas-api" Jul 07 16:08:08.577540 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:08.577472 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3" containerName="maas-api" Jul 07 16:08:08.577613 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:08.577551 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a93680b-3f9b-4ba8-9fd9-dfe11aafcbe3" containerName="maas-api" Jul 07 16:08:08.580861 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:08.580841 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54" Jul 07 16:08:08.585378 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:08.585351 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-30f6fc42-serving-cert\"" Jul 07 16:08:08.596453 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:08.596423 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54"] Jul 07 16:08:08.651564 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:08.651494 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpl9r\" (UniqueName: \"kubernetes.io/projected/ef07a371-c55d-4169-a7f9-3c31a89c3dd4-kube-api-access-bpl9r\") pod \"maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54\" (UID: \"ef07a371-c55d-4169-a7f9-3c31a89c3dd4\") " pod="opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54" Jul 07 16:08:08.651756 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:08.651591 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef07a371-c55d-4169-a7f9-3c31a89c3dd4-maas-api-tls\") pod \"maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54\" (UID: \"ef07a371-c55d-4169-a7f9-3c31a89c3dd4\") " pod="opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54" Jul 07 16:08:08.651756 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:08.651708 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef07a371-c55d-4169-a7f9-3c31a89c3dd4-openshift-service-ca\") pod \"maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54\" (UID: \"ef07a371-c55d-4169-a7f9-3c31a89c3dd4\") " pod="opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54" Jul 07 16:08:08.752989 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:08.752952 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpl9r\" (UniqueName: \"kubernetes.io/projected/ef07a371-c55d-4169-a7f9-3c31a89c3dd4-kube-api-access-bpl9r\") pod \"maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54\" (UID: \"ef07a371-c55d-4169-a7f9-3c31a89c3dd4\") " pod="opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54" Jul 07 16:08:08.752989 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:08.752997 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef07a371-c55d-4169-a7f9-3c31a89c3dd4-maas-api-tls\") pod \"maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54\" (UID: \"ef07a371-c55d-4169-a7f9-3c31a89c3dd4\") " pod="opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54" Jul 07 16:08:08.753252 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:08.753070 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef07a371-c55d-4169-a7f9-3c31a89c3dd4-openshift-service-ca\") pod \"maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54\" (UID: \"ef07a371-c55d-4169-a7f9-3c31a89c3dd4\") " pod="opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54" Jul 07 16:08:08.753836 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:08.753806 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef07a371-c55d-4169-a7f9-3c31a89c3dd4-openshift-service-ca\") pod \"maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54\" (UID: \"ef07a371-c55d-4169-a7f9-3c31a89c3dd4\") " pod="opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54" Jul 07 16:08:08.755591 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:08.755568 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef07a371-c55d-4169-a7f9-3c31a89c3dd4-maas-api-tls\") pod \"maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54\" (UID: \"ef07a371-c55d-4169-a7f9-3c31a89c3dd4\") " pod="opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54" Jul 07 16:08:08.767244 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:08.767215 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpl9r\" (UniqueName: \"kubernetes.io/projected/ef07a371-c55d-4169-a7f9-3c31a89c3dd4-kube-api-access-bpl9r\") pod \"maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54\" (UID: \"ef07a371-c55d-4169-a7f9-3c31a89c3dd4\") " pod="opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54" Jul 07 16:08:08.892228 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:08.892136 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54" Jul 07 16:08:09.049083 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:09.049055 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54"] Jul 07 16:08:09.050309 ip-10-0-139-248 kubenswrapper[2557]: W0707 16:08:09.050285 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef07a371_c55d_4169_a7f9_3c31a89c3dd4.slice/crio-28b9ded52e4f459a16a7c49dfc1b49bf67df31106c55cf480b4693fa77335477 WatchSource:0}: Error finding container 28b9ded52e4f459a16a7c49dfc1b49bf67df31106c55cf480b4693fa77335477: Status 404 returned error can't find the container with id 28b9ded52e4f459a16a7c49dfc1b49bf67df31106c55cf480b4693fa77335477 Jul 07 16:08:09.100780 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:09.100749 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54" event={"ID":"ef07a371-c55d-4169-a7f9-3c31a89c3dd4","Type":"ContainerStarted","Data":"28b9ded52e4f459a16a7c49dfc1b49bf67df31106c55cf480b4693fa77335477"} Jul 07 16:08:10.108052 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:10.108009 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54" event={"ID":"ef07a371-c55d-4169-a7f9-3c31a89c3dd4","Type":"ContainerStarted","Data":"2c6367952434b7fa3e1498702400bfee4540d4fae98ed4dc26aec838f56c8b3c"} Jul 07 16:08:10.108470 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:10.108183 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54" Jul 07 16:08:10.127054 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:10.126995 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54" podStartSLOduration=1.7558928219999999 podStartE2EDuration="2.126978016s" totalImagesPullingTime="371.085194ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:08:10.12431276 +0000 UTC m=+1674.301957210" watchObservedRunningTime="2026-07-07 16:08:10.126978016 +0000 UTC m=+1674.304622466" Jul 07 16:08:16.120275 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:16.120243 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54" Jul 07 16:08:56.159956 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:56.159873 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54"] Jul 07 16:08:56.160456 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:56.160100 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54" podUID="ef07a371-c55d-4169-a7f9-3c31a89c3dd4" containerName="maas-api" containerID="cri-o://2c6367952434b7fa3e1498702400bfee4540d4fae98ed4dc26aec838f56c8b3c" gracePeriod=30 Jul 07 16:08:56.382977 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:56.382938 2557 generic.go:356] "Generic (PLEG): container finished" podID="ef07a371-c55d-4169-a7f9-3c31a89c3dd4" containerID="2c6367952434b7fa3e1498702400bfee4540d4fae98ed4dc26aec838f56c8b3c" exitCode=0 Jul 07 16:08:56.383164 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:56.383005 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54" event={"ID":"ef07a371-c55d-4169-a7f9-3c31a89c3dd4","Type":"ContainerDied","Data":"2c6367952434b7fa3e1498702400bfee4540d4fae98ed4dc26aec838f56c8b3c"} Jul 07 16:08:56.418546 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:56.418484 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54" Jul 07 16:08:56.504395 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:56.504361 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpl9r\" (UniqueName: \"kubernetes.io/projected/ef07a371-c55d-4169-a7f9-3c31a89c3dd4-kube-api-access-bpl9r\") pod \"ef07a371-c55d-4169-a7f9-3c31a89c3dd4\" (UID: \"ef07a371-c55d-4169-a7f9-3c31a89c3dd4\") " Jul 07 16:08:56.504597 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:56.504450 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef07a371-c55d-4169-a7f9-3c31a89c3dd4-maas-api-tls\") pod \"ef07a371-c55d-4169-a7f9-3c31a89c3dd4\" (UID: \"ef07a371-c55d-4169-a7f9-3c31a89c3dd4\") " Jul 07 16:08:56.504597 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:56.504491 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef07a371-c55d-4169-a7f9-3c31a89c3dd4-openshift-service-ca\") pod \"ef07a371-c55d-4169-a7f9-3c31a89c3dd4\" (UID: \"ef07a371-c55d-4169-a7f9-3c31a89c3dd4\") " Jul 07 16:08:56.504901 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:56.504878 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef07a371-c55d-4169-a7f9-3c31a89c3dd4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ef07a371-c55d-4169-a7f9-3c31a89c3dd4" (UID: "ef07a371-c55d-4169-a7f9-3c31a89c3dd4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:08:56.506652 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:56.506626 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef07a371-c55d-4169-a7f9-3c31a89c3dd4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ef07a371-c55d-4169-a7f9-3c31a89c3dd4" (UID: "ef07a371-c55d-4169-a7f9-3c31a89c3dd4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:08:56.506755 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:56.506633 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef07a371-c55d-4169-a7f9-3c31a89c3dd4-kube-api-access-bpl9r" (OuterVolumeSpecName: "kube-api-access-bpl9r") pod "ef07a371-c55d-4169-a7f9-3c31a89c3dd4" (UID: "ef07a371-c55d-4169-a7f9-3c31a89c3dd4"). InnerVolumeSpecName "kube-api-access-bpl9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:08:56.606212 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:56.606181 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpl9r\" (UniqueName: \"kubernetes.io/projected/ef07a371-c55d-4169-a7f9-3c31a89c3dd4-kube-api-access-bpl9r\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:08:56.606212 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:56.606207 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef07a371-c55d-4169-a7f9-3c31a89c3dd4-maas-api-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:08:56.606212 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:56.606217 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef07a371-c55d-4169-a7f9-3c31a89c3dd4-openshift-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:08:57.390133 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:57.390094 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54" Jul 07 16:08:57.390625 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:57.390148 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54" event={"ID":"ef07a371-c55d-4169-a7f9-3c31a89c3dd4","Type":"ContainerDied","Data":"28b9ded52e4f459a16a7c49dfc1b49bf67df31106c55cf480b4693fa77335477"} Jul 07 16:08:57.390625 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:57.390212 2557 scope.go:117] "RemoveContainer" containerID="2c6367952434b7fa3e1498702400bfee4540d4fae98ed4dc26aec838f56c8b3c" Jul 07 16:08:57.413818 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:57.413782 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54"] Jul 07 16:08:57.417498 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:57.417473 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-30f6fc42-6bcff5fb4b-z4w54"] Jul 07 16:08:58.351535 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:08:58.351499 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef07a371-c55d-4169-a7f9-3c31a89c3dd4" path="/var/lib/kubelet/pods/ef07a371-c55d-4169-a7f9-3c31a89c3dd4/volumes" Jul 07 16:10:16.461290 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:16.461261 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/1.log" Jul 07 16:10:16.465952 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:16.465928 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hkj2z_07a80436-65b9-4f0d-99b3-ecda36240b48/ovn-acl-logging/0.log" Jul 07 16:10:16.466083 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:16.465962 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/1.log" Jul 07 16:10:16.470654 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:16.470627 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hkj2z_07a80436-65b9-4f0d-99b3-ecda36240b48/ovn-acl-logging/0.log" Jul 07 16:10:48.561005 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:48.560972 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s"] Jul 07 16:10:48.561498 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:48.561441 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef07a371-c55d-4169-a7f9-3c31a89c3dd4" containerName="maas-api" Jul 07 16:10:48.561498 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:48.561454 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef07a371-c55d-4169-a7f9-3c31a89c3dd4" containerName="maas-api" Jul 07 16:10:48.561576 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:48.561516 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef07a371-c55d-4169-a7f9-3c31a89c3dd4" containerName="maas-api" Jul 07 16:10:48.565244 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:48.565221 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" Jul 07 16:10:48.567313 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:48.567290 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-e6fe85-serving-cert\"" Jul 07 16:10:48.581259 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:48.581229 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s"] Jul 07 16:10:48.638762 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:48.638731 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8qw8\" (UniqueName: \"kubernetes.io/projected/759edfd1-d9aa-4362-925c-61488515cc44-kube-api-access-g8qw8\") pod \"maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s\" (UID: \"759edfd1-d9aa-4362-925c-61488515cc44\") " pod="opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" Jul 07 16:10:48.638973 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:48.638793 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/759edfd1-d9aa-4362-925c-61488515cc44-maas-api-tls\") pod \"maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s\" (UID: \"759edfd1-d9aa-4362-925c-61488515cc44\") " pod="opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" Jul 07 16:10:48.638973 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:48.638821 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/759edfd1-d9aa-4362-925c-61488515cc44-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s\" (UID: \"759edfd1-d9aa-4362-925c-61488515cc44\") " pod="opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" Jul 07 16:10:48.739712 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:48.739669 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8qw8\" (UniqueName: \"kubernetes.io/projected/759edfd1-d9aa-4362-925c-61488515cc44-kube-api-access-g8qw8\") pod \"maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s\" (UID: \"759edfd1-d9aa-4362-925c-61488515cc44\") " pod="opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" Jul 07 16:10:48.739923 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:48.739730 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/759edfd1-d9aa-4362-925c-61488515cc44-maas-api-tls\") pod \"maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s\" (UID: \"759edfd1-d9aa-4362-925c-61488515cc44\") " pod="opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" Jul 07 16:10:48.739923 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:48.739765 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/759edfd1-d9aa-4362-925c-61488515cc44-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s\" (UID: \"759edfd1-d9aa-4362-925c-61488515cc44\") " pod="opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" Jul 07 16:10:48.739923 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:10:48.739875 2557 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-e6fe85-serving-cert: secret "maas-api-e2e-inf-a-e6fe85-serving-cert" not found Jul 07 16:10:48.740100 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:10:48.739968 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/759edfd1-d9aa-4362-925c-61488515cc44-maas-api-tls podName:759edfd1-d9aa-4362-925c-61488515cc44 nodeName:}" failed. No retries permitted until 2026-07-07 16:10:49.239934749 +0000 UTC m=+1833.417579192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/759edfd1-d9aa-4362-925c-61488515cc44-maas-api-tls") pod "maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" (UID: "759edfd1-d9aa-4362-925c-61488515cc44") : secret "maas-api-e2e-inf-a-e6fe85-serving-cert" not found Jul 07 16:10:48.740473 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:48.740455 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/759edfd1-d9aa-4362-925c-61488515cc44-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s\" (UID: \"759edfd1-d9aa-4362-925c-61488515cc44\") " pod="opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" Jul 07 16:10:48.751596 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:48.751572 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8qw8\" (UniqueName: \"kubernetes.io/projected/759edfd1-d9aa-4362-925c-61488515cc44-kube-api-access-g8qw8\") pod \"maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s\" (UID: \"759edfd1-d9aa-4362-925c-61488515cc44\") " pod="opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" Jul 07 16:10:49.245590 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:49.245551 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/759edfd1-d9aa-4362-925c-61488515cc44-maas-api-tls\") pod \"maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s\" (UID: \"759edfd1-d9aa-4362-925c-61488515cc44\") " pod="opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" Jul 07 16:10:49.248290 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:49.248264 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/759edfd1-d9aa-4362-925c-61488515cc44-maas-api-tls\") pod \"maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s\" (UID: \"759edfd1-d9aa-4362-925c-61488515cc44\") " pod="opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" Jul 07 16:10:49.479099 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:49.479065 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" Jul 07 16:10:49.635333 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:49.635301 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s"] Jul 07 16:10:49.636754 ip-10-0-139-248 kubenswrapper[2557]: W0707 16:10:49.636709 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod759edfd1_d9aa_4362_925c_61488515cc44.slice/crio-ef83326a7325f3a3df5cb71ebee87cad7370240e4a8a1b7575403989ee15df52 WatchSource:0}: Error finding container ef83326a7325f3a3df5cb71ebee87cad7370240e4a8a1b7575403989ee15df52: Status 404 returned error can't find the container with id ef83326a7325f3a3df5cb71ebee87cad7370240e4a8a1b7575403989ee15df52 Jul 07 16:10:49.638177 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:49.638158 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 16:10:50.047530 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:50.047498 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" event={"ID":"759edfd1-d9aa-4362-925c-61488515cc44","Type":"ContainerStarted","Data":"ef83326a7325f3a3df5cb71ebee87cad7370240e4a8a1b7575403989ee15df52"} Jul 07 16:10:51.054711 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:51.054669 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" event={"ID":"759edfd1-d9aa-4362-925c-61488515cc44","Type":"ContainerStarted","Data":"42269799c4912050c1947005fd850c274ed410212b7725b2ee2705cc2cf38b9e"} Jul 07 16:10:51.055177 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:51.054755 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" Jul 07 16:10:51.077661 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:51.077612 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" podStartSLOduration=2.62087259 podStartE2EDuration="3.077594586s" totalImagesPullingTime="456.721996ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:10:51.076860463 +0000 UTC m=+1835.254504914" watchObservedRunningTime="2026-07-07 16:10:51.077594586 +0000 UTC m=+1835.255239038" Jul 07 16:10:56.389927 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:56.389873 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7"] Jul 07 16:10:56.394477 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:56.394449 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7" Jul 07 16:10:56.396647 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:56.396619 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-11d09f-serving-cert\"" Jul 07 16:10:56.403057 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:56.403029 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7"] Jul 07 16:10:56.515047 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:56.515017 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c29c63b-7002-4cc6-acee-1f0be0ed9696-maas-api-tls\") pod \"maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7\" (UID: \"7c29c63b-7002-4cc6-acee-1f0be0ed9696\") " pod="opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7" Jul 07 16:10:56.515265 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:56.515161 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phrg5\" (UniqueName: \"kubernetes.io/projected/7c29c63b-7002-4cc6-acee-1f0be0ed9696-kube-api-access-phrg5\") pod \"maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7\" (UID: \"7c29c63b-7002-4cc6-acee-1f0be0ed9696\") " pod="opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7" Jul 07 16:10:56.515265 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:56.515194 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c29c63b-7002-4cc6-acee-1f0be0ed9696-openshift-service-ca\") pod \"maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7\" (UID: \"7c29c63b-7002-4cc6-acee-1f0be0ed9696\") " pod="opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7" Jul 07 16:10:56.615993 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:56.615956 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c29c63b-7002-4cc6-acee-1f0be0ed9696-openshift-service-ca\") pod \"maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7\" (UID: \"7c29c63b-7002-4cc6-acee-1f0be0ed9696\") " pod="opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7" Jul 07 16:10:56.616221 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:56.616005 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c29c63b-7002-4cc6-acee-1f0be0ed9696-maas-api-tls\") pod \"maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7\" (UID: \"7c29c63b-7002-4cc6-acee-1f0be0ed9696\") " pod="opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7" Jul 07 16:10:56.616221 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:56.616076 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phrg5\" (UniqueName: \"kubernetes.io/projected/7c29c63b-7002-4cc6-acee-1f0be0ed9696-kube-api-access-phrg5\") pod \"maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7\" (UID: \"7c29c63b-7002-4cc6-acee-1f0be0ed9696\") " pod="opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7" Jul 07 16:10:56.616682 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:56.616660 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c29c63b-7002-4cc6-acee-1f0be0ed9696-openshift-service-ca\") pod \"maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7\" (UID: \"7c29c63b-7002-4cc6-acee-1f0be0ed9696\") " pod="opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7" Jul 07 16:10:56.618560 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:56.618538 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c29c63b-7002-4cc6-acee-1f0be0ed9696-maas-api-tls\") pod \"maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7\" (UID: \"7c29c63b-7002-4cc6-acee-1f0be0ed9696\") " pod="opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7" Jul 07 16:10:56.624320 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:56.624300 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phrg5\" (UniqueName: \"kubernetes.io/projected/7c29c63b-7002-4cc6-acee-1f0be0ed9696-kube-api-access-phrg5\") pod \"maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7\" (UID: \"7c29c63b-7002-4cc6-acee-1f0be0ed9696\") " pod="opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7" Jul 07 16:10:56.708904 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:56.708822 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7" Jul 07 16:10:57.061966 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:57.061925 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7"] Jul 07 16:10:57.063536 ip-10-0-139-248 kubenswrapper[2557]: W0707 16:10:57.063509 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c29c63b_7002_4cc6_acee_1f0be0ed9696.slice/crio-ce043d54755539090babe208e20e345a09c22c1660c2fa892ac911444fc3212f WatchSource:0}: Error finding container ce043d54755539090babe208e20e345a09c22c1660c2fa892ac911444fc3212f: Status 404 returned error can't find the container with id ce043d54755539090babe208e20e345a09c22c1660c2fa892ac911444fc3212f Jul 07 16:10:57.068646 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:57.068625 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" Jul 07 16:10:57.092673 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:57.092644 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7" event={"ID":"7c29c63b-7002-4cc6-acee-1f0be0ed9696","Type":"ContainerStarted","Data":"ce043d54755539090babe208e20e345a09c22c1660c2fa892ac911444fc3212f"} Jul 07 16:10:58.100758 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:58.100721 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7" event={"ID":"7c29c63b-7002-4cc6-acee-1f0be0ed9696","Type":"ContainerStarted","Data":"ce607f5df07c70a309cbbf4c19f4cb85e05e998f7d96644b67ea71a4a30a7f42"} Jul 07 16:10:58.101279 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:58.100858 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7" Jul 07 16:10:58.126601 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:10:58.126547 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7" podStartSLOduration=1.68406795 podStartE2EDuration="2.126526026s" totalImagesPullingTime="442.458076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:10:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:10:58.126213535 +0000 UTC m=+1842.303858003" watchObservedRunningTime="2026-07-07 16:10:58.126526026 +0000 UTC m=+1842.304170477" Jul 07 16:11:04.112910 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:11:04.112878 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7" Jul 07 16:12:01.586270 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:01.586169 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s"] Jul 07 16:12:01.586769 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:01.586425 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" podUID="759edfd1-d9aa-4362-925c-61488515cc44" containerName="maas-api" containerID="cri-o://42269799c4912050c1947005fd850c274ed410212b7725b2ee2705cc2cf38b9e" gracePeriod=30 Jul 07 16:12:01.840388 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:01.840328 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" Jul 07 16:12:01.931746 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:01.931714 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/759edfd1-d9aa-4362-925c-61488515cc44-maas-api-tls\") pod \"759edfd1-d9aa-4362-925c-61488515cc44\" (UID: \"759edfd1-d9aa-4362-925c-61488515cc44\") " Jul 07 16:12:01.931932 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:01.931835 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8qw8\" (UniqueName: \"kubernetes.io/projected/759edfd1-d9aa-4362-925c-61488515cc44-kube-api-access-g8qw8\") pod \"759edfd1-d9aa-4362-925c-61488515cc44\" (UID: \"759edfd1-d9aa-4362-925c-61488515cc44\") " Jul 07 16:12:01.931932 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:01.931856 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/759edfd1-d9aa-4362-925c-61488515cc44-openshift-service-ca\") pod \"759edfd1-d9aa-4362-925c-61488515cc44\" (UID: \"759edfd1-d9aa-4362-925c-61488515cc44\") " Jul 07 16:12:01.932338 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:01.932313 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/759edfd1-d9aa-4362-925c-61488515cc44-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "759edfd1-d9aa-4362-925c-61488515cc44" (UID: "759edfd1-d9aa-4362-925c-61488515cc44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:12:01.933859 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:01.933834 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/759edfd1-d9aa-4362-925c-61488515cc44-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "759edfd1-d9aa-4362-925c-61488515cc44" (UID: "759edfd1-d9aa-4362-925c-61488515cc44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:12:01.933999 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:01.933976 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759edfd1-d9aa-4362-925c-61488515cc44-kube-api-access-g8qw8" (OuterVolumeSpecName: "kube-api-access-g8qw8") pod "759edfd1-d9aa-4362-925c-61488515cc44" (UID: "759edfd1-d9aa-4362-925c-61488515cc44"). InnerVolumeSpecName "kube-api-access-g8qw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:12:02.032538 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:02.032507 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8qw8\" (UniqueName: \"kubernetes.io/projected/759edfd1-d9aa-4362-925c-61488515cc44-kube-api-access-g8qw8\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:12:02.032538 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:02.032535 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/759edfd1-d9aa-4362-925c-61488515cc44-openshift-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:12:02.032727 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:02.032547 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/759edfd1-d9aa-4362-925c-61488515cc44-maas-api-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:12:02.481506 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:02.481467 2557 generic.go:356] "Generic (PLEG): container finished" podID="759edfd1-d9aa-4362-925c-61488515cc44" containerID="42269799c4912050c1947005fd850c274ed410212b7725b2ee2705cc2cf38b9e" exitCode=0 Jul 07 16:12:02.481697 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:02.481533 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" Jul 07 16:12:02.481697 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:02.481555 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" event={"ID":"759edfd1-d9aa-4362-925c-61488515cc44","Type":"ContainerDied","Data":"42269799c4912050c1947005fd850c274ed410212b7725b2ee2705cc2cf38b9e"} Jul 07 16:12:02.481697 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:02.481634 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s" event={"ID":"759edfd1-d9aa-4362-925c-61488515cc44","Type":"ContainerDied","Data":"ef83326a7325f3a3df5cb71ebee87cad7370240e4a8a1b7575403989ee15df52"} Jul 07 16:12:02.481697 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:02.481674 2557 scope.go:117] "RemoveContainer" containerID="42269799c4912050c1947005fd850c274ed410212b7725b2ee2705cc2cf38b9e" Jul 07 16:12:02.491611 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:02.491596 2557 scope.go:117] "RemoveContainer" containerID="42269799c4912050c1947005fd850c274ed410212b7725b2ee2705cc2cf38b9e" Jul 07 16:12:02.491867 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:12:02.491846 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42269799c4912050c1947005fd850c274ed410212b7725b2ee2705cc2cf38b9e\": container with ID starting with 42269799c4912050c1947005fd850c274ed410212b7725b2ee2705cc2cf38b9e not found: ID does not exist" containerID="42269799c4912050c1947005fd850c274ed410212b7725b2ee2705cc2cf38b9e" Jul 07 16:12:02.491916 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:02.491875 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42269799c4912050c1947005fd850c274ed410212b7725b2ee2705cc2cf38b9e"} err="failed to get container status \"42269799c4912050c1947005fd850c274ed410212b7725b2ee2705cc2cf38b9e\": rpc error: code = NotFound desc = could not find container \"42269799c4912050c1947005fd850c274ed410212b7725b2ee2705cc2cf38b9e\": container with ID starting with 42269799c4912050c1947005fd850c274ed410212b7725b2ee2705cc2cf38b9e not found: ID does not exist" Jul 07 16:12:02.497192 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:02.497172 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s"] Jul 07 16:12:02.501255 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:02.501234 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-e6fe85-666fbf6979-c7h4s"] Jul 07 16:12:04.352071 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:04.352039 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="759edfd1-d9aa-4362-925c-61488515cc44" path="/var/lib/kubelet/pods/759edfd1-d9aa-4362-925c-61488515cc44/volumes" Jul 07 16:12:44.608792 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:44.608758 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7"] Jul 07 16:12:44.609443 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:44.609053 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7" podUID="7c29c63b-7002-4cc6-acee-1f0be0ed9696" containerName="maas-api" containerID="cri-o://ce607f5df07c70a309cbbf4c19f4cb85e05e998f7d96644b67ea71a4a30a7f42" gracePeriod=30 Jul 07 16:12:44.737693 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:44.737661 2557 generic.go:356] "Generic (PLEG): container finished" podID="7c29c63b-7002-4cc6-acee-1f0be0ed9696" containerID="ce607f5df07c70a309cbbf4c19f4cb85e05e998f7d96644b67ea71a4a30a7f42" exitCode=0 Jul 07 16:12:44.737840 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:44.737724 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7" event={"ID":"7c29c63b-7002-4cc6-acee-1f0be0ed9696","Type":"ContainerDied","Data":"ce607f5df07c70a309cbbf4c19f4cb85e05e998f7d96644b67ea71a4a30a7f42"} Jul 07 16:12:44.862299 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:44.862237 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7" Jul 07 16:12:44.926737 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:44.926704 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c29c63b-7002-4cc6-acee-1f0be0ed9696-openshift-service-ca\") pod \"7c29c63b-7002-4cc6-acee-1f0be0ed9696\" (UID: \"7c29c63b-7002-4cc6-acee-1f0be0ed9696\") " Jul 07 16:12:44.926907 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:44.926790 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c29c63b-7002-4cc6-acee-1f0be0ed9696-maas-api-tls\") pod \"7c29c63b-7002-4cc6-acee-1f0be0ed9696\" (UID: \"7c29c63b-7002-4cc6-acee-1f0be0ed9696\") " Jul 07 16:12:44.926907 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:44.926861 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phrg5\" (UniqueName: \"kubernetes.io/projected/7c29c63b-7002-4cc6-acee-1f0be0ed9696-kube-api-access-phrg5\") pod \"7c29c63b-7002-4cc6-acee-1f0be0ed9696\" (UID: \"7c29c63b-7002-4cc6-acee-1f0be0ed9696\") " Jul 07 16:12:44.927156 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:44.927132 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c29c63b-7002-4cc6-acee-1f0be0ed9696-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7c29c63b-7002-4cc6-acee-1f0be0ed9696" (UID: "7c29c63b-7002-4cc6-acee-1f0be0ed9696"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:12:44.928968 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:44.928939 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c29c63b-7002-4cc6-acee-1f0be0ed9696-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7c29c63b-7002-4cc6-acee-1f0be0ed9696" (UID: "7c29c63b-7002-4cc6-acee-1f0be0ed9696"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:12:44.929080 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:44.928983 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c29c63b-7002-4cc6-acee-1f0be0ed9696-kube-api-access-phrg5" (OuterVolumeSpecName: "kube-api-access-phrg5") pod "7c29c63b-7002-4cc6-acee-1f0be0ed9696" (UID: "7c29c63b-7002-4cc6-acee-1f0be0ed9696"). InnerVolumeSpecName "kube-api-access-phrg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:12:45.027834 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:45.027793 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c29c63b-7002-4cc6-acee-1f0be0ed9696-openshift-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:12:45.027834 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:45.027827 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c29c63b-7002-4cc6-acee-1f0be0ed9696-maas-api-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:12:45.027834 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:45.027837 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phrg5\" (UniqueName: \"kubernetes.io/projected/7c29c63b-7002-4cc6-acee-1f0be0ed9696-kube-api-access-phrg5\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:12:45.745297 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:45.745269 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7" Jul 07 16:12:45.745719 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:45.745310 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7" event={"ID":"7c29c63b-7002-4cc6-acee-1f0be0ed9696","Type":"ContainerDied","Data":"ce043d54755539090babe208e20e345a09c22c1660c2fa892ac911444fc3212f"} Jul 07 16:12:45.745719 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:45.745368 2557 scope.go:117] "RemoveContainer" containerID="ce607f5df07c70a309cbbf4c19f4cb85e05e998f7d96644b67ea71a4a30a7f42" Jul 07 16:12:45.768002 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:45.767979 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7"] Jul 07 16:12:45.773398 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:45.773373 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-11d09f-7956678df4-k5fd7"] Jul 07 16:12:46.351775 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:46.351741 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c29c63b-7002-4cc6-acee-1f0be0ed9696" path="/var/lib/kubelet/pods/7c29c63b-7002-4cc6-acee-1f0be0ed9696/volumes" Jul 07 16:12:54.068405 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:54.068371 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f"] Jul 07 16:12:54.068846 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:54.068812 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c29c63b-7002-4cc6-acee-1f0be0ed9696" containerName="maas-api" Jul 07 16:12:54.068846 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:54.068826 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c29c63b-7002-4cc6-acee-1f0be0ed9696" containerName="maas-api" Jul 07 16:12:54.068846 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:54.068836 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="759edfd1-d9aa-4362-925c-61488515cc44" containerName="maas-api" Jul 07 16:12:54.068846 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:54.068841 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="759edfd1-d9aa-4362-925c-61488515cc44" containerName="maas-api" Jul 07 16:12:54.068983 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:54.068918 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c29c63b-7002-4cc6-acee-1f0be0ed9696" containerName="maas-api" Jul 07 16:12:54.068983 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:54.068929 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="759edfd1-d9aa-4362-925c-61488515cc44" containerName="maas-api" Jul 07 16:12:54.072247 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:54.072219 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f" Jul 07 16:12:54.074388 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:54.074360 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-0aba26-serving-cert\"" Jul 07 16:12:54.091805 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:54.091776 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f"] Jul 07 16:12:54.215806 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:54.215766 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae5e465-8488-4807-9310-4fe63c6b0f82-openshift-service-ca\") pod \"maas-api-e2e-api-a-0aba26-5cc476754f-dm65f\" (UID: \"4ae5e465-8488-4807-9310-4fe63c6b0f82\") " pod="opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f" Jul 07 16:12:54.216010 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:54.215838 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mcp8\" (UniqueName: \"kubernetes.io/projected/4ae5e465-8488-4807-9310-4fe63c6b0f82-kube-api-access-4mcp8\") pod \"maas-api-e2e-api-a-0aba26-5cc476754f-dm65f\" (UID: \"4ae5e465-8488-4807-9310-4fe63c6b0f82\") " pod="opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f" Jul 07 16:12:54.216010 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:54.215906 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ae5e465-8488-4807-9310-4fe63c6b0f82-maas-api-tls\") pod \"maas-api-e2e-api-a-0aba26-5cc476754f-dm65f\" (UID: \"4ae5e465-8488-4807-9310-4fe63c6b0f82\") " pod="opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f" Jul 07 16:12:54.316495 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:54.316459 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae5e465-8488-4807-9310-4fe63c6b0f82-openshift-service-ca\") pod \"maas-api-e2e-api-a-0aba26-5cc476754f-dm65f\" (UID: \"4ae5e465-8488-4807-9310-4fe63c6b0f82\") " pod="opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f" Jul 07 16:12:54.316706 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:54.316508 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mcp8\" (UniqueName: \"kubernetes.io/projected/4ae5e465-8488-4807-9310-4fe63c6b0f82-kube-api-access-4mcp8\") pod \"maas-api-e2e-api-a-0aba26-5cc476754f-dm65f\" (UID: \"4ae5e465-8488-4807-9310-4fe63c6b0f82\") " pod="opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f" Jul 07 16:12:54.316706 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:54.316545 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ae5e465-8488-4807-9310-4fe63c6b0f82-maas-api-tls\") pod \"maas-api-e2e-api-a-0aba26-5cc476754f-dm65f\" (UID: \"4ae5e465-8488-4807-9310-4fe63c6b0f82\") " pod="opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f" Jul 07 16:12:54.317238 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:54.317213 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae5e465-8488-4807-9310-4fe63c6b0f82-openshift-service-ca\") pod \"maas-api-e2e-api-a-0aba26-5cc476754f-dm65f\" (UID: \"4ae5e465-8488-4807-9310-4fe63c6b0f82\") " pod="opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f" Jul 07 16:12:54.319031 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:54.318982 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ae5e465-8488-4807-9310-4fe63c6b0f82-maas-api-tls\") pod \"maas-api-e2e-api-a-0aba26-5cc476754f-dm65f\" (UID: \"4ae5e465-8488-4807-9310-4fe63c6b0f82\") " pod="opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f" Jul 07 16:12:54.324074 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:54.324053 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mcp8\" (UniqueName: \"kubernetes.io/projected/4ae5e465-8488-4807-9310-4fe63c6b0f82-kube-api-access-4mcp8\") pod \"maas-api-e2e-api-a-0aba26-5cc476754f-dm65f\" (UID: \"4ae5e465-8488-4807-9310-4fe63c6b0f82\") " pod="opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f" Jul 07 16:12:54.384505 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:54.384476 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f" Jul 07 16:12:54.534036 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:54.534002 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f"] Jul 07 16:12:54.535736 ip-10-0-139-248 kubenswrapper[2557]: W0707 16:12:54.535711 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae5e465_8488_4807_9310_4fe63c6b0f82.slice/crio-e070bacc7c258e8ebc3f4248c636ed5aac4124eaa985f17a7e667e69a3373837 WatchSource:0}: Error finding container e070bacc7c258e8ebc3f4248c636ed5aac4124eaa985f17a7e667e69a3373837: Status 404 returned error can't find the container with id e070bacc7c258e8ebc3f4248c636ed5aac4124eaa985f17a7e667e69a3373837 Jul 07 16:12:54.802755 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:54.802717 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f" event={"ID":"4ae5e465-8488-4807-9310-4fe63c6b0f82","Type":"ContainerStarted","Data":"e070bacc7c258e8ebc3f4248c636ed5aac4124eaa985f17a7e667e69a3373837"} Jul 07 16:12:55.810402 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:55.810369 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f" event={"ID":"4ae5e465-8488-4807-9310-4fe63c6b0f82","Type":"ContainerStarted","Data":"0d1f625736a0c6060b3debe214afe37bee1db28b55d671dfce1c15640b34b5e8"} Jul 07 16:12:55.810858 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:55.810492 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f" Jul 07 16:12:55.829918 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:12:55.829865 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f" podStartSLOduration=1.210395213 podStartE2EDuration="1.829850733s" totalImagesPullingTime="619.45552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:12:55.827056438 +0000 UTC m=+1960.004700911" watchObservedRunningTime="2026-07-07 16:12:55.829850733 +0000 UTC m=+1960.007495182" Jul 07 16:13:01.823051 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:01.823019 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f" Jul 07 16:13:02.072797 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:02.072760 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v"] Jul 07 16:13:02.076837 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:02.076769 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" Jul 07 16:13:02.079444 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:02.079147 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-7820dd-serving-cert\"" Jul 07 16:13:02.091891 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:02.091865 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v"] Jul 07 16:13:02.188222 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:02.188174 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbwpw\" (UniqueName: \"kubernetes.io/projected/8177f1d1-e6ef-4e67-ac14-c7693bb071ba-kube-api-access-mbwpw\") pod \"maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v\" (UID: \"8177f1d1-e6ef-4e67-ac14-c7693bb071ba\") " pod="opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" Jul 07 16:13:02.188378 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:02.188289 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8177f1d1-e6ef-4e67-ac14-c7693bb071ba-openshift-service-ca\") pod \"maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v\" (UID: \"8177f1d1-e6ef-4e67-ac14-c7693bb071ba\") " pod="opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" Jul 07 16:13:02.188378 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:02.188353 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8177f1d1-e6ef-4e67-ac14-c7693bb071ba-maas-api-tls\") pod \"maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v\" (UID: \"8177f1d1-e6ef-4e67-ac14-c7693bb071ba\") " pod="opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" Jul 07 16:13:02.289292 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:02.289250 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8177f1d1-e6ef-4e67-ac14-c7693bb071ba-openshift-service-ca\") pod \"maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v\" (UID: \"8177f1d1-e6ef-4e67-ac14-c7693bb071ba\") " pod="opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" Jul 07 16:13:02.289489 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:02.289316 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8177f1d1-e6ef-4e67-ac14-c7693bb071ba-maas-api-tls\") pod \"maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v\" (UID: \"8177f1d1-e6ef-4e67-ac14-c7693bb071ba\") " pod="opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" Jul 07 16:13:02.289489 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:02.289415 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwpw\" (UniqueName: \"kubernetes.io/projected/8177f1d1-e6ef-4e67-ac14-c7693bb071ba-kube-api-access-mbwpw\") pod \"maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v\" (UID: \"8177f1d1-e6ef-4e67-ac14-c7693bb071ba\") " pod="opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" Jul 07 16:13:02.289489 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:13:02.289478 2557 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-7820dd-serving-cert: secret "maas-api-e2e-api-b-7820dd-serving-cert" not found Jul 07 16:13:02.289683 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:13:02.289595 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8177f1d1-e6ef-4e67-ac14-c7693bb071ba-maas-api-tls podName:8177f1d1-e6ef-4e67-ac14-c7693bb071ba nodeName:}" failed. No retries permitted until 2026-07-07 16:13:02.789573601 +0000 UTC m=+1966.967218041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8177f1d1-e6ef-4e67-ac14-c7693bb071ba-maas-api-tls") pod "maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" (UID: "8177f1d1-e6ef-4e67-ac14-c7693bb071ba") : secret "maas-api-e2e-api-b-7820dd-serving-cert" not found Jul 07 16:13:02.290182 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:02.290159 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8177f1d1-e6ef-4e67-ac14-c7693bb071ba-openshift-service-ca\") pod \"maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v\" (UID: \"8177f1d1-e6ef-4e67-ac14-c7693bb071ba\") " pod="opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" Jul 07 16:13:02.309470 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:02.309431 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwpw\" (UniqueName: \"kubernetes.io/projected/8177f1d1-e6ef-4e67-ac14-c7693bb071ba-kube-api-access-mbwpw\") pod \"maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v\" (UID: \"8177f1d1-e6ef-4e67-ac14-c7693bb071ba\") " pod="opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" Jul 07 16:13:02.795933 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:02.795893 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8177f1d1-e6ef-4e67-ac14-c7693bb071ba-maas-api-tls\") pod \"maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v\" (UID: \"8177f1d1-e6ef-4e67-ac14-c7693bb071ba\") " pod="opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" Jul 07 16:13:02.798693 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:02.798656 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8177f1d1-e6ef-4e67-ac14-c7693bb071ba-maas-api-tls\") pod \"maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v\" (UID: \"8177f1d1-e6ef-4e67-ac14-c7693bb071ba\") " pod="opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" Jul 07 16:13:02.989087 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:02.989046 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" Jul 07 16:13:03.156924 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:03.156896 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v"] Jul 07 16:13:03.158297 ip-10-0-139-248 kubenswrapper[2557]: W0707 16:13:03.158257 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8177f1d1_e6ef_4e67_ac14_c7693bb071ba.slice/crio-b61ae462c56ca406a0e858742fee35b3240baae56975e71fb58c3873da3df88e WatchSource:0}: Error finding container b61ae462c56ca406a0e858742fee35b3240baae56975e71fb58c3873da3df88e: Status 404 returned error can't find the container with id b61ae462c56ca406a0e858742fee35b3240baae56975e71fb58c3873da3df88e Jul 07 16:13:03.861505 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:03.861470 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" event={"ID":"8177f1d1-e6ef-4e67-ac14-c7693bb071ba","Type":"ContainerStarted","Data":"7c4ea18783b004a4fdcdca99d47474c5513f397752dffb999abc7455d1309890"} Jul 07 16:13:03.861668 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:03.861529 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" event={"ID":"8177f1d1-e6ef-4e67-ac14-c7693bb071ba","Type":"ContainerStarted","Data":"b61ae462c56ca406a0e858742fee35b3240baae56975e71fb58c3873da3df88e"} Jul 07 16:13:03.861668 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:03.861651 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" Jul 07 16:13:03.880996 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:03.880950 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" podStartSLOduration=1.291551939 podStartE2EDuration="1.88093525s" totalImagesPullingTime="589.383311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:13:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:13:03.878596757 +0000 UTC m=+1968.056241207" watchObservedRunningTime="2026-07-07 16:13:03.88093525 +0000 UTC m=+1968.058579700" Jul 07 16:13:09.873224 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:09.873108 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" Jul 07 16:13:15.429966 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:15.429932 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f"] Jul 07 16:13:15.430497 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:15.430202 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f" podUID="4ae5e465-8488-4807-9310-4fe63c6b0f82" containerName="maas-api" containerID="cri-o://0d1f625736a0c6060b3debe214afe37bee1db28b55d671dfce1c15640b34b5e8" gracePeriod=30 Jul 07 16:13:15.695817 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:15.695749 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f" Jul 07 16:13:15.724501 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:15.724469 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ae5e465-8488-4807-9310-4fe63c6b0f82-maas-api-tls\") pod \"4ae5e465-8488-4807-9310-4fe63c6b0f82\" (UID: \"4ae5e465-8488-4807-9310-4fe63c6b0f82\") " Jul 07 16:13:15.724687 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:15.724522 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mcp8\" (UniqueName: \"kubernetes.io/projected/4ae5e465-8488-4807-9310-4fe63c6b0f82-kube-api-access-4mcp8\") pod \"4ae5e465-8488-4807-9310-4fe63c6b0f82\" (UID: \"4ae5e465-8488-4807-9310-4fe63c6b0f82\") " Jul 07 16:13:15.724687 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:15.724610 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae5e465-8488-4807-9310-4fe63c6b0f82-openshift-service-ca\") pod \"4ae5e465-8488-4807-9310-4fe63c6b0f82\" (UID: \"4ae5e465-8488-4807-9310-4fe63c6b0f82\") " Jul 07 16:13:15.724954 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:15.724928 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae5e465-8488-4807-9310-4fe63c6b0f82-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4ae5e465-8488-4807-9310-4fe63c6b0f82" (UID: "4ae5e465-8488-4807-9310-4fe63c6b0f82"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:13:15.726758 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:15.726732 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae5e465-8488-4807-9310-4fe63c6b0f82-kube-api-access-4mcp8" (OuterVolumeSpecName: "kube-api-access-4mcp8") pod "4ae5e465-8488-4807-9310-4fe63c6b0f82" (UID: "4ae5e465-8488-4807-9310-4fe63c6b0f82"). InnerVolumeSpecName "kube-api-access-4mcp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:13:15.726865 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:15.726767 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ae5e465-8488-4807-9310-4fe63c6b0f82-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4ae5e465-8488-4807-9310-4fe63c6b0f82" (UID: "4ae5e465-8488-4807-9310-4fe63c6b0f82"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:13:15.826014 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:15.825979 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ae5e465-8488-4807-9310-4fe63c6b0f82-maas-api-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:13:15.826014 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:15.826008 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mcp8\" (UniqueName: \"kubernetes.io/projected/4ae5e465-8488-4807-9310-4fe63c6b0f82-kube-api-access-4mcp8\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:13:15.826014 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:15.826018 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae5e465-8488-4807-9310-4fe63c6b0f82-openshift-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:13:15.936013 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:15.935974 2557 generic.go:356] "Generic (PLEG): container finished" podID="4ae5e465-8488-4807-9310-4fe63c6b0f82" containerID="0d1f625736a0c6060b3debe214afe37bee1db28b55d671dfce1c15640b34b5e8" exitCode=0 Jul 07 16:13:15.936185 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:15.936024 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f" Jul 07 16:13:15.936185 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:15.936058 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f" event={"ID":"4ae5e465-8488-4807-9310-4fe63c6b0f82","Type":"ContainerDied","Data":"0d1f625736a0c6060b3debe214afe37bee1db28b55d671dfce1c15640b34b5e8"} Jul 07 16:13:15.936185 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:15.936130 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f" event={"ID":"4ae5e465-8488-4807-9310-4fe63c6b0f82","Type":"ContainerDied","Data":"e070bacc7c258e8ebc3f4248c636ed5aac4124eaa985f17a7e667e69a3373837"} Jul 07 16:13:15.936185 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:15.936160 2557 scope.go:117] "RemoveContainer" containerID="0d1f625736a0c6060b3debe214afe37bee1db28b55d671dfce1c15640b34b5e8" Jul 07 16:13:15.946067 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:15.946050 2557 scope.go:117] "RemoveContainer" containerID="0d1f625736a0c6060b3debe214afe37bee1db28b55d671dfce1c15640b34b5e8" Jul 07 16:13:15.946357 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:13:15.946339 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d1f625736a0c6060b3debe214afe37bee1db28b55d671dfce1c15640b34b5e8\": container with ID starting with 0d1f625736a0c6060b3debe214afe37bee1db28b55d671dfce1c15640b34b5e8 not found: ID does not exist" containerID="0d1f625736a0c6060b3debe214afe37bee1db28b55d671dfce1c15640b34b5e8" Jul 07 16:13:15.946441 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:15.946367 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1f625736a0c6060b3debe214afe37bee1db28b55d671dfce1c15640b34b5e8"} err="failed to get container status \"0d1f625736a0c6060b3debe214afe37bee1db28b55d671dfce1c15640b34b5e8\": rpc error: code = NotFound desc = could not find container \"0d1f625736a0c6060b3debe214afe37bee1db28b55d671dfce1c15640b34b5e8\": container with ID starting with 0d1f625736a0c6060b3debe214afe37bee1db28b55d671dfce1c15640b34b5e8 not found: ID does not exist" Jul 07 16:13:15.959067 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:15.959036 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f"] Jul 07 16:13:15.962214 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:15.962190 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-0aba26-5cc476754f-dm65f"] Jul 07 16:13:16.351968 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:16.351938 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ae5e465-8488-4807-9310-4fe63c6b0f82" path="/var/lib/kubelet/pods/4ae5e465-8488-4807-9310-4fe63c6b0f82/volumes" Jul 07 16:13:23.231030 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:23.230994 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v"] Jul 07 16:13:23.231484 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:23.231257 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" podUID="8177f1d1-e6ef-4e67-ac14-c7693bb071ba" containerName="maas-api" containerID="cri-o://7c4ea18783b004a4fdcdca99d47474c5513f397752dffb999abc7455d1309890" gracePeriod=30 Jul 07 16:13:23.486564 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:23.486497 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" Jul 07 16:13:23.596453 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:23.596419 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8177f1d1-e6ef-4e67-ac14-c7693bb071ba-openshift-service-ca\") pod \"8177f1d1-e6ef-4e67-ac14-c7693bb071ba\" (UID: \"8177f1d1-e6ef-4e67-ac14-c7693bb071ba\") " Jul 07 16:13:23.596621 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:23.596509 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8177f1d1-e6ef-4e67-ac14-c7693bb071ba-maas-api-tls\") pod \"8177f1d1-e6ef-4e67-ac14-c7693bb071ba\" (UID: \"8177f1d1-e6ef-4e67-ac14-c7693bb071ba\") " Jul 07 16:13:23.596621 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:23.596556 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbwpw\" (UniqueName: \"kubernetes.io/projected/8177f1d1-e6ef-4e67-ac14-c7693bb071ba-kube-api-access-mbwpw\") pod \"8177f1d1-e6ef-4e67-ac14-c7693bb071ba\" (UID: \"8177f1d1-e6ef-4e67-ac14-c7693bb071ba\") " Jul 07 16:13:23.596866 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:23.596838 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8177f1d1-e6ef-4e67-ac14-c7693bb071ba-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8177f1d1-e6ef-4e67-ac14-c7693bb071ba" (UID: "8177f1d1-e6ef-4e67-ac14-c7693bb071ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:13:23.598691 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:23.598662 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8177f1d1-e6ef-4e67-ac14-c7693bb071ba-kube-api-access-mbwpw" (OuterVolumeSpecName: "kube-api-access-mbwpw") pod "8177f1d1-e6ef-4e67-ac14-c7693bb071ba" (UID: "8177f1d1-e6ef-4e67-ac14-c7693bb071ba"). InnerVolumeSpecName "kube-api-access-mbwpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:13:23.598817 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:23.598726 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8177f1d1-e6ef-4e67-ac14-c7693bb071ba-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8177f1d1-e6ef-4e67-ac14-c7693bb071ba" (UID: "8177f1d1-e6ef-4e67-ac14-c7693bb071ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:13:23.698099 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:23.698059 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8177f1d1-e6ef-4e67-ac14-c7693bb071ba-openshift-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:13:23.698099 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:23.698092 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8177f1d1-e6ef-4e67-ac14-c7693bb071ba-maas-api-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:13:23.698099 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:23.698102 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbwpw\" (UniqueName: \"kubernetes.io/projected/8177f1d1-e6ef-4e67-ac14-c7693bb071ba-kube-api-access-mbwpw\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:13:23.993652 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:23.993619 2557 generic.go:356] "Generic (PLEG): container finished" podID="8177f1d1-e6ef-4e67-ac14-c7693bb071ba" containerID="7c4ea18783b004a4fdcdca99d47474c5513f397752dffb999abc7455d1309890" exitCode=0 Jul 07 16:13:23.993843 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:23.993669 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" Jul 07 16:13:23.993843 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:23.993711 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" event={"ID":"8177f1d1-e6ef-4e67-ac14-c7693bb071ba","Type":"ContainerDied","Data":"7c4ea18783b004a4fdcdca99d47474c5513f397752dffb999abc7455d1309890"} Jul 07 16:13:23.993843 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:23.993770 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v" event={"ID":"8177f1d1-e6ef-4e67-ac14-c7693bb071ba","Type":"ContainerDied","Data":"b61ae462c56ca406a0e858742fee35b3240baae56975e71fb58c3873da3df88e"} Jul 07 16:13:23.993843 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:23.993794 2557 scope.go:117] "RemoveContainer" containerID="7c4ea18783b004a4fdcdca99d47474c5513f397752dffb999abc7455d1309890" Jul 07 16:13:24.003520 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:24.003505 2557 scope.go:117] "RemoveContainer" containerID="7c4ea18783b004a4fdcdca99d47474c5513f397752dffb999abc7455d1309890" Jul 07 16:13:24.003758 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:13:24.003739 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c4ea18783b004a4fdcdca99d47474c5513f397752dffb999abc7455d1309890\": container with ID starting with 7c4ea18783b004a4fdcdca99d47474c5513f397752dffb999abc7455d1309890 not found: ID does not exist" containerID="7c4ea18783b004a4fdcdca99d47474c5513f397752dffb999abc7455d1309890" Jul 07 16:13:24.003812 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:24.003763 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c4ea18783b004a4fdcdca99d47474c5513f397752dffb999abc7455d1309890"} err="failed to get container status \"7c4ea18783b004a4fdcdca99d47474c5513f397752dffb999abc7455d1309890\": rpc error: code = NotFound desc = could not find container \"7c4ea18783b004a4fdcdca99d47474c5513f397752dffb999abc7455d1309890\": container with ID starting with 7c4ea18783b004a4fdcdca99d47474c5513f397752dffb999abc7455d1309890 not found: ID does not exist" Jul 07 16:13:24.014700 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:24.014676 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v"] Jul 07 16:13:24.017712 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:24.017688 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-7820dd-5f4dc48dc7-w7t7v"] Jul 07 16:13:24.352528 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:24.352495 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8177f1d1-e6ef-4e67-ac14-c7693bb071ba" path="/var/lib/kubelet/pods/8177f1d1-e6ef-4e67-ac14-c7693bb071ba/volumes" Jul 07 16:13:33.324121 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:33.324072 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh"] Jul 07 16:13:33.324578 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:33.324543 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ae5e465-8488-4807-9310-4fe63c6b0f82" containerName="maas-api" Jul 07 16:13:33.324578 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:33.324557 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ae5e465-8488-4807-9310-4fe63c6b0f82" containerName="maas-api" Jul 07 16:13:33.324578 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:33.324568 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8177f1d1-e6ef-4e67-ac14-c7693bb071ba" containerName="maas-api" Jul 07 16:13:33.324578 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:33.324576 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8177f1d1-e6ef-4e67-ac14-c7693bb071ba" containerName="maas-api" Jul 07 16:13:33.324713 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:33.324643 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8177f1d1-e6ef-4e67-ac14-c7693bb071ba" containerName="maas-api" Jul 07 16:13:33.324713 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:33.324652 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ae5e465-8488-4807-9310-4fe63c6b0f82" containerName="maas-api" Jul 07 16:13:33.329280 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:33.329258 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" Jul 07 16:13:33.331429 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:33.331410 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-12f2df-serving-cert\"" Jul 07 16:13:33.345692 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:33.345663 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh"] Jul 07 16:13:33.398179 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:33.398138 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmz68\" (UniqueName: \"kubernetes.io/projected/d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d-kube-api-access-gmz68\") pod \"maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh\" (UID: \"d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d\") " pod="opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" Jul 07 16:13:33.398342 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:33.398262 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh\" (UID: \"d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d\") " pod="opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" Jul 07 16:13:33.398342 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:33.398309 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d-maas-api-tls\") pod \"maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh\" (UID: \"d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d\") " pod="opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" Jul 07 16:13:33.499158 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:33.499106 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh\" (UID: \"d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d\") " pod="opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" Jul 07 16:13:33.499368 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:33.499180 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d-maas-api-tls\") pod \"maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh\" (UID: \"d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d\") " pod="opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" Jul 07 16:13:33.499368 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:33.499236 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmz68\" (UniqueName: \"kubernetes.io/projected/d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d-kube-api-access-gmz68\") pod \"maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh\" (UID: \"d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d\") " pod="opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" Jul 07 16:13:33.499368 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:13:33.499342 2557 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-12f2df-serving-cert: secret "maas-api-e2e-shared-a-12f2df-serving-cert" not found Jul 07 16:13:33.499537 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:13:33.499427 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d-maas-api-tls podName:d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d nodeName:}" failed. No retries permitted until 2026-07-07 16:13:33.999403337 +0000 UTC m=+1998.177047765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d-maas-api-tls") pod "maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" (UID: "d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d") : secret "maas-api-e2e-shared-a-12f2df-serving-cert" not found Jul 07 16:13:33.499856 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:33.499833 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh\" (UID: \"d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d\") " pod="opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" Jul 07 16:13:33.507572 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:33.507549 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmz68\" (UniqueName: \"kubernetes.io/projected/d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d-kube-api-access-gmz68\") pod \"maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh\" (UID: \"d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d\") " pod="opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" Jul 07 16:13:34.003721 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:34.003680 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d-maas-api-tls\") pod \"maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh\" (UID: \"d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d\") " pod="opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" Jul 07 16:13:34.006386 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:34.006351 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d-maas-api-tls\") pod \"maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh\" (UID: \"d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d\") " pod="opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" Jul 07 16:13:34.241961 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:34.241867 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" Jul 07 16:13:34.393782 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:34.393666 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh"] Jul 07 16:13:34.396704 ip-10-0-139-248 kubenswrapper[2557]: W0707 16:13:34.396677 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ad92e6_1a34_4b36_bd2b_a507e8ebe50d.slice/crio-b184227bd4ab42efd24107b947fd4eee9f17e435ad60d576454afb81e2959346 WatchSource:0}: Error finding container b184227bd4ab42efd24107b947fd4eee9f17e435ad60d576454afb81e2959346: Status 404 returned error can't find the container with id b184227bd4ab42efd24107b947fd4eee9f17e435ad60d576454afb81e2959346 Jul 07 16:13:35.060605 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:35.060567 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" event={"ID":"d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d","Type":"ContainerStarted","Data":"9185e1c28a38bff33865039bb9e81d64068f2136e97c2b7f1837dcf3b6e12133"} Jul 07 16:13:35.060816 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:35.060616 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" event={"ID":"d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d","Type":"ContainerStarted","Data":"b184227bd4ab42efd24107b947fd4eee9f17e435ad60d576454afb81e2959346"} Jul 07 16:13:35.060816 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:35.060774 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" Jul 07 16:13:35.079263 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:35.079202 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" podStartSLOduration=1.56267731 podStartE2EDuration="2.07918311s" totalImagesPullingTime="516.5058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:13:35.076939369 +0000 UTC m=+1999.254583820" watchObservedRunningTime="2026-07-07 16:13:35.07918311 +0000 UTC m=+1999.256827560" Jul 07 16:13:41.073291 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:41.073253 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" Jul 07 16:13:41.312387 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:41.312353 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln"] Jul 07 16:13:41.316240 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:41.316218 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" Jul 07 16:13:41.318609 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:41.318585 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-ac6096-serving-cert\"" Jul 07 16:13:41.330855 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:41.330830 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln"] Jul 07 16:13:41.477096 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:41.477054 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln\" (UID: \"0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8\") " pod="opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" Jul 07 16:13:41.477096 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:41.477096 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whmvp\" (UniqueName: \"kubernetes.io/projected/0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8-kube-api-access-whmvp\") pod \"maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln\" (UID: \"0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8\") " pod="opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" Jul 07 16:13:41.477381 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:41.477222 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln\" (UID: \"0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8\") " pod="opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" Jul 07 16:13:41.578331 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:41.578232 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln\" (UID: \"0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8\") " pod="opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" Jul 07 16:13:41.578331 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:41.578288 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whmvp\" (UniqueName: \"kubernetes.io/projected/0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8-kube-api-access-whmvp\") pod \"maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln\" (UID: \"0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8\") " pod="opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" Jul 07 16:13:41.578569 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:41.578335 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln\" (UID: \"0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8\") " pod="opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" Jul 07 16:13:41.578569 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:13:41.578509 2557 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-ac6096-serving-cert: secret "maas-api-e2e-shared-b-ac6096-serving-cert" not found Jul 07 16:13:41.578684 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:13:41.578588 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8-maas-api-tls podName:0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8 nodeName:}" failed. No retries permitted until 2026-07-07 16:13:42.07856716 +0000 UTC m=+2006.256211591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8-maas-api-tls") pod "maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" (UID: "0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8") : secret "maas-api-e2e-shared-b-ac6096-serving-cert" not found Jul 07 16:13:41.578955 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:41.578935 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln\" (UID: \"0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8\") " pod="opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" Jul 07 16:13:41.586677 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:41.586650 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whmvp\" (UniqueName: \"kubernetes.io/projected/0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8-kube-api-access-whmvp\") pod \"maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln\" (UID: \"0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8\") " pod="opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" Jul 07 16:13:42.082680 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:42.082648 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln\" (UID: \"0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8\") " pod="opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" Jul 07 16:13:42.085555 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:42.085522 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln\" (UID: \"0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8\") " pod="opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" Jul 07 16:13:42.228277 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:42.228240 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" Jul 07 16:13:42.580900 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:42.580871 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln"] Jul 07 16:13:42.583152 ip-10-0-139-248 kubenswrapper[2557]: W0707 16:13:42.583091 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0512faf5_78bb_4ba1_bdd7_0bf1cb4ff2d8.slice/crio-4022ba13873b1aa4496c0df942ff82be2b53ae302e938b051a0ab325c043c6ea WatchSource:0}: Error finding container 4022ba13873b1aa4496c0df942ff82be2b53ae302e938b051a0ab325c043c6ea: Status 404 returned error can't find the container with id 4022ba13873b1aa4496c0df942ff82be2b53ae302e938b051a0ab325c043c6ea Jul 07 16:13:43.110710 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:43.110679 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" event={"ID":"0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8","Type":"ContainerStarted","Data":"9d9970966c72739b7efabbfca8bbfb63535abcba4c70881d6e4a9200934922d6"} Jul 07 16:13:43.111099 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:43.110736 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" event={"ID":"0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8","Type":"ContainerStarted","Data":"4022ba13873b1aa4496c0df942ff82be2b53ae302e938b051a0ab325c043c6ea"} Jul 07 16:13:43.111099 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:43.110835 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" Jul 07 16:13:43.131423 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:43.131372 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" podStartSLOduration=1.696451664 podStartE2EDuration="2.131357646s" totalImagesPullingTime="434.905982ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:13:43.129726324 +0000 UTC m=+2007.307370774" watchObservedRunningTime="2026-07-07 16:13:43.131357646 +0000 UTC m=+2007.309002095" Jul 07 16:13:49.122001 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:13:49.121969 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" Jul 07 16:15:00.143060 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.142964 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-12f2df-29724015-rt8v7"] Jul 07 16:15:00.146620 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.146596 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29724015-mtwsp"] Jul 07 16:15:00.147178 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.146717 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-12f2df-29724015-rt8v7" Jul 07 16:15:00.150970 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.150945 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-ac6096-29724015-z4qvp"] Jul 07 16:15:00.151094 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.151038 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724015-mtwsp" Jul 07 16:15:00.155515 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.155491 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-ac6096-29724015-z4qvp" Jul 07 16:15:00.156714 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.156696 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-ac6096-29724015-z4qvp"] Jul 07 16:15:00.160472 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.160440 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-12f2df-29724015-rt8v7"] Jul 07 16:15:00.164271 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.164250 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724015-mtwsp"] Jul 07 16:15:00.198578 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.198548 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ldjn\" (UniqueName: \"kubernetes.io/projected/890454b7-b54c-4796-b587-b3b8b9e0306c-kube-api-access-4ldjn\") pod \"maas-api-key-cleanup-e2e-shared-b-ac6096-29724015-z4qvp\" (UID: \"890454b7-b54c-4796-b587-b3b8b9e0306c\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-ac6096-29724015-z4qvp" Jul 07 16:15:00.198703 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.198622 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blpbl\" (UniqueName: \"kubernetes.io/projected/a4aa7bee-c3b4-4c91-815a-dd67e527684e-kube-api-access-blpbl\") pod \"maas-api-key-cleanup-e2e-shared-a-12f2df-29724015-rt8v7\" (UID: \"a4aa7bee-c3b4-4c91-815a-dd67e527684e\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-12f2df-29724015-rt8v7" Jul 07 16:15:00.198703 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.198677 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4q5v\" (UniqueName: \"kubernetes.io/projected/77b65a9e-fb4f-49c2-be4e-0c463535d64d-kube-api-access-x4q5v\") pod \"maas-api-key-cleanup-29724015-mtwsp\" (UID: \"77b65a9e-fb4f-49c2-be4e-0c463535d64d\") " pod="opendatahub/maas-api-key-cleanup-29724015-mtwsp" Jul 07 16:15:00.299964 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.299929 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ldjn\" (UniqueName: \"kubernetes.io/projected/890454b7-b54c-4796-b587-b3b8b9e0306c-kube-api-access-4ldjn\") pod \"maas-api-key-cleanup-e2e-shared-b-ac6096-29724015-z4qvp\" (UID: \"890454b7-b54c-4796-b587-b3b8b9e0306c\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-ac6096-29724015-z4qvp" Jul 07 16:15:00.300155 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.299982 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blpbl\" (UniqueName: \"kubernetes.io/projected/a4aa7bee-c3b4-4c91-815a-dd67e527684e-kube-api-access-blpbl\") pod \"maas-api-key-cleanup-e2e-shared-a-12f2df-29724015-rt8v7\" (UID: \"a4aa7bee-c3b4-4c91-815a-dd67e527684e\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-12f2df-29724015-rt8v7" Jul 07 16:15:00.300155 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.300013 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4q5v\" (UniqueName: \"kubernetes.io/projected/77b65a9e-fb4f-49c2-be4e-0c463535d64d-kube-api-access-x4q5v\") pod \"maas-api-key-cleanup-29724015-mtwsp\" (UID: \"77b65a9e-fb4f-49c2-be4e-0c463535d64d\") " pod="opendatahub/maas-api-key-cleanup-29724015-mtwsp" Jul 07 16:15:00.308875 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.308848 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4q5v\" (UniqueName: \"kubernetes.io/projected/77b65a9e-fb4f-49c2-be4e-0c463535d64d-kube-api-access-x4q5v\") pod \"maas-api-key-cleanup-29724015-mtwsp\" (UID: \"77b65a9e-fb4f-49c2-be4e-0c463535d64d\") " pod="opendatahub/maas-api-key-cleanup-29724015-mtwsp" Jul 07 16:15:00.309183 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.309160 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blpbl\" (UniqueName: \"kubernetes.io/projected/a4aa7bee-c3b4-4c91-815a-dd67e527684e-kube-api-access-blpbl\") pod \"maas-api-key-cleanup-e2e-shared-a-12f2df-29724015-rt8v7\" (UID: \"a4aa7bee-c3b4-4c91-815a-dd67e527684e\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-12f2df-29724015-rt8v7" Jul 07 16:15:00.309239 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.309160 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ldjn\" (UniqueName: \"kubernetes.io/projected/890454b7-b54c-4796-b587-b3b8b9e0306c-kube-api-access-4ldjn\") pod \"maas-api-key-cleanup-e2e-shared-b-ac6096-29724015-z4qvp\" (UID: \"890454b7-b54c-4796-b587-b3b8b9e0306c\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-ac6096-29724015-z4qvp" Jul 07 16:15:00.462269 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.462189 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-12f2df-29724015-rt8v7" Jul 07 16:15:00.472232 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.472207 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724015-mtwsp" Jul 07 16:15:00.481330 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.481305 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-ac6096-29724015-z4qvp" Jul 07 16:15:00.639630 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.639598 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-12f2df-29724015-rt8v7"] Jul 07 16:15:00.642458 ip-10-0-139-248 kubenswrapper[2557]: W0707 16:15:00.642418 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4aa7bee_c3b4_4c91_815a_dd67e527684e.slice/crio-2f2dc7d6a280a36425ca1e38b6e25b67bb393abeb5e8380b870618941b073590 WatchSource:0}: Error finding container 2f2dc7d6a280a36425ca1e38b6e25b67bb393abeb5e8380b870618941b073590: Status 404 returned error can't find the container with id 2f2dc7d6a280a36425ca1e38b6e25b67bb393abeb5e8380b870618941b073590 Jul 07 16:15:00.866317 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.866287 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724015-mtwsp"] Jul 07 16:15:00.867363 ip-10-0-139-248 kubenswrapper[2557]: W0707 16:15:00.867323 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b65a9e_fb4f_49c2_be4e_0c463535d64d.slice/crio-1d6c66a2166709f6f1acb9e56f49791ad6c25f4848b2ee2c4920a03c53a12146 WatchSource:0}: Error finding container 1d6c66a2166709f6f1acb9e56f49791ad6c25f4848b2ee2c4920a03c53a12146: Status 404 returned error can't find the container with id 1d6c66a2166709f6f1acb9e56f49791ad6c25f4848b2ee2c4920a03c53a12146 Jul 07 16:15:00.897035 ip-10-0-139-248 kubenswrapper[2557]: W0707 16:15:00.897004 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890454b7_b54c_4796_b587_b3b8b9e0306c.slice/crio-4692815cfa3627270017a0792762bd7a2155cb6efd4d1c12686588aa0494c7a0 WatchSource:0}: Error finding container 4692815cfa3627270017a0792762bd7a2155cb6efd4d1c12686588aa0494c7a0: Status 404 returned error can't find the container with id 4692815cfa3627270017a0792762bd7a2155cb6efd4d1c12686588aa0494c7a0 Jul 07 16:15:00.897035 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:00.897008 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-ac6096-29724015-z4qvp"] Jul 07 16:15:01.592575 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:01.592540 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-12f2df-29724015-rt8v7" event={"ID":"a4aa7bee-c3b4-4c91-815a-dd67e527684e","Type":"ContainerStarted","Data":"109f3a5a8b08c90b4237c77f0e3718fdf04e306881b85b256e2668f64825ad27"} Jul 07 16:15:01.593015 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:01.592599 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-12f2df-29724015-rt8v7" event={"ID":"a4aa7bee-c3b4-4c91-815a-dd67e527684e","Type":"ContainerStarted","Data":"2f2dc7d6a280a36425ca1e38b6e25b67bb393abeb5e8380b870618941b073590"} Jul 07 16:15:01.594149 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:01.594105 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-ac6096-29724015-z4qvp" event={"ID":"890454b7-b54c-4796-b587-b3b8b9e0306c","Type":"ContainerStarted","Data":"226f2810e23ef5cfcabfbbd7b64a9ac449787e9a6670303a5fb982cef7e51220"} Jul 07 16:15:01.594248 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:01.594174 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-ac6096-29724015-z4qvp" event={"ID":"890454b7-b54c-4796-b587-b3b8b9e0306c","Type":"ContainerStarted","Data":"4692815cfa3627270017a0792762bd7a2155cb6efd4d1c12686588aa0494c7a0"} Jul 07 16:15:01.595690 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:01.595662 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724015-mtwsp" event={"ID":"77b65a9e-fb4f-49c2-be4e-0c463535d64d","Type":"ContainerStarted","Data":"84873f621becaf46bac673f8d2a5c455265e64ec40d18b54a7bed9287625f2c4"} Jul 07 16:15:01.595784 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:01.595709 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724015-mtwsp" event={"ID":"77b65a9e-fb4f-49c2-be4e-0c463535d64d","Type":"ContainerStarted","Data":"1d6c66a2166709f6f1acb9e56f49791ad6c25f4848b2ee2c4920a03c53a12146"} Jul 07 16:15:01.610328 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:01.610281 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-12f2df-29724015-rt8v7" podStartSLOduration=1.610267884 podStartE2EDuration="1.610267884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:15:01.60794516 +0000 UTC m=+2085.785589612" watchObservedRunningTime="2026-07-07 16:15:01.610267884 +0000 UTC m=+2085.787912333" Jul 07 16:15:01.625554 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:01.625512 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29724015-mtwsp" podStartSLOduration=1.625500072 podStartE2EDuration="1.625500072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:15:01.623107125 +0000 UTC m=+2085.800751574" watchObservedRunningTime="2026-07-07 16:15:01.625500072 +0000 UTC m=+2085.803144521" Jul 07 16:15:01.644618 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:01.644577 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-ac6096-29724015-z4qvp" podStartSLOduration=1.64456542 podStartE2EDuration="1.64456542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:15:01.641540548 +0000 UTC m=+2085.819184998" watchObservedRunningTime="2026-07-07 16:15:01.64456542 +0000 UTC m=+2085.822209870" Jul 07 16:15:02.603282 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:02.603253 2557 generic.go:356] "Generic (PLEG): container finished" podID="a4aa7bee-c3b4-4c91-815a-dd67e527684e" containerID="109f3a5a8b08c90b4237c77f0e3718fdf04e306881b85b256e2668f64825ad27" exitCode=0 Jul 07 16:15:02.603710 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:02.603331 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-12f2df-29724015-rt8v7" event={"ID":"a4aa7bee-c3b4-4c91-815a-dd67e527684e","Type":"ContainerDied","Data":"109f3a5a8b08c90b4237c77f0e3718fdf04e306881b85b256e2668f64825ad27"} Jul 07 16:15:02.604888 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:02.604864 2557 generic.go:356] "Generic (PLEG): container finished" podID="890454b7-b54c-4796-b587-b3b8b9e0306c" containerID="226f2810e23ef5cfcabfbbd7b64a9ac449787e9a6670303a5fb982cef7e51220" exitCode=0 Jul 07 16:15:02.604967 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:02.604942 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-ac6096-29724015-z4qvp" event={"ID":"890454b7-b54c-4796-b587-b3b8b9e0306c","Type":"ContainerDied","Data":"226f2810e23ef5cfcabfbbd7b64a9ac449787e9a6670303a5fb982cef7e51220"} Jul 07 16:15:02.606487 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:02.606467 2557 generic.go:356] "Generic (PLEG): container finished" podID="77b65a9e-fb4f-49c2-be4e-0c463535d64d" containerID="84873f621becaf46bac673f8d2a5c455265e64ec40d18b54a7bed9287625f2c4" exitCode=0 Jul 07 16:15:02.606581 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:02.606541 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724015-mtwsp" event={"ID":"77b65a9e-fb4f-49c2-be4e-0c463535d64d","Type":"ContainerDied","Data":"84873f621becaf46bac673f8d2a5c455265e64ec40d18b54a7bed9287625f2c4"} Jul 07 16:15:03.780752 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:03.780726 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-12f2df-29724015-rt8v7" Jul 07 16:15:03.833246 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:03.833221 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blpbl\" (UniqueName: \"kubernetes.io/projected/a4aa7bee-c3b4-4c91-815a-dd67e527684e-kube-api-access-blpbl\") pod \"a4aa7bee-c3b4-4c91-815a-dd67e527684e\" (UID: \"a4aa7bee-c3b4-4c91-815a-dd67e527684e\") " Jul 07 16:15:03.835403 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:03.835379 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4aa7bee-c3b4-4c91-815a-dd67e527684e-kube-api-access-blpbl" (OuterVolumeSpecName: "kube-api-access-blpbl") pod "a4aa7bee-c3b4-4c91-815a-dd67e527684e" (UID: "a4aa7bee-c3b4-4c91-815a-dd67e527684e"). InnerVolumeSpecName "kube-api-access-blpbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:15:03.837584 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:03.837564 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724015-mtwsp" Jul 07 16:15:03.847768 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:03.847749 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-ac6096-29724015-z4qvp" Jul 07 16:15:03.934264 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:03.934194 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ldjn\" (UniqueName: \"kubernetes.io/projected/890454b7-b54c-4796-b587-b3b8b9e0306c-kube-api-access-4ldjn\") pod \"890454b7-b54c-4796-b587-b3b8b9e0306c\" (UID: \"890454b7-b54c-4796-b587-b3b8b9e0306c\") " Jul 07 16:15:03.934433 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:03.934296 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4q5v\" (UniqueName: \"kubernetes.io/projected/77b65a9e-fb4f-49c2-be4e-0c463535d64d-kube-api-access-x4q5v\") pod \"77b65a9e-fb4f-49c2-be4e-0c463535d64d\" (UID: \"77b65a9e-fb4f-49c2-be4e-0c463535d64d\") " Jul 07 16:15:03.934516 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:03.934498 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blpbl\" (UniqueName: \"kubernetes.io/projected/a4aa7bee-c3b4-4c91-815a-dd67e527684e-kube-api-access-blpbl\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:15:03.936095 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:03.936071 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/890454b7-b54c-4796-b587-b3b8b9e0306c-kube-api-access-4ldjn" (OuterVolumeSpecName: "kube-api-access-4ldjn") pod "890454b7-b54c-4796-b587-b3b8b9e0306c" (UID: "890454b7-b54c-4796-b587-b3b8b9e0306c"). InnerVolumeSpecName "kube-api-access-4ldjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:15:03.936218 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:03.936171 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b65a9e-fb4f-49c2-be4e-0c463535d64d-kube-api-access-x4q5v" (OuterVolumeSpecName: "kube-api-access-x4q5v") pod "77b65a9e-fb4f-49c2-be4e-0c463535d64d" (UID: "77b65a9e-fb4f-49c2-be4e-0c463535d64d"). InnerVolumeSpecName "kube-api-access-x4q5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:15:04.035775 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:04.035737 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4q5v\" (UniqueName: \"kubernetes.io/projected/77b65a9e-fb4f-49c2-be4e-0c463535d64d-kube-api-access-x4q5v\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:15:04.035775 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:04.035770 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ldjn\" (UniqueName: \"kubernetes.io/projected/890454b7-b54c-4796-b587-b3b8b9e0306c-kube-api-access-4ldjn\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:15:04.620713 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:04.620626 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724015-mtwsp" Jul 07 16:15:04.620882 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:04.620624 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724015-mtwsp" event={"ID":"77b65a9e-fb4f-49c2-be4e-0c463535d64d","Type":"ContainerDied","Data":"1d6c66a2166709f6f1acb9e56f49791ad6c25f4848b2ee2c4920a03c53a12146"} Jul 07 16:15:04.620882 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:04.620760 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d6c66a2166709f6f1acb9e56f49791ad6c25f4848b2ee2c4920a03c53a12146" Jul 07 16:15:04.622031 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:04.622012 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-12f2df-29724015-rt8v7" Jul 07 16:15:04.622228 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:04.622042 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-12f2df-29724015-rt8v7" event={"ID":"a4aa7bee-c3b4-4c91-815a-dd67e527684e","Type":"ContainerDied","Data":"2f2dc7d6a280a36425ca1e38b6e25b67bb393abeb5e8380b870618941b073590"} Jul 07 16:15:04.622228 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:04.622085 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f2dc7d6a280a36425ca1e38b6e25b67bb393abeb5e8380b870618941b073590" Jul 07 16:15:04.623519 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:04.623503 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-ac6096-29724015-z4qvp" Jul 07 16:15:04.623519 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:04.623512 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-ac6096-29724015-z4qvp" event={"ID":"890454b7-b54c-4796-b587-b3b8b9e0306c","Type":"ContainerDied","Data":"4692815cfa3627270017a0792762bd7a2155cb6efd4d1c12686588aa0494c7a0"} Jul 07 16:15:04.623654 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:04.623541 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4692815cfa3627270017a0792762bd7a2155cb6efd4d1c12686588aa0494c7a0" Jul 07 16:15:16.502804 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:16.502774 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/1.log" Jul 07 16:15:16.507375 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:16.507349 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hkj2z_07a80436-65b9-4f0d-99b3-ecda36240b48/ovn-acl-logging/0.log" Jul 07 16:15:16.508940 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:16.508918 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/1.log" Jul 07 16:15:16.513651 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:15:16.513634 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hkj2z_07a80436-65b9-4f0d-99b3-ecda36240b48/ovn-acl-logging/0.log" Jul 07 16:19:17.670561 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:17.670467 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh"] Jul 07 16:19:17.671182 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:17.670735 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" podUID="d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d" containerName="maas-api" containerID="cri-o://9185e1c28a38bff33865039bb9e81d64068f2136e97c2b7f1837dcf3b6e12133" gracePeriod=30 Jul 07 16:19:17.710852 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:17.710815 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-12f2df-29724015-rt8v7"] Jul 07 16:19:17.713518 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:17.713494 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-12f2df-29724015-rt8v7"] Jul 07 16:19:17.933761 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:17.933693 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" Jul 07 16:19:18.100009 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:18.099978 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d-maas-api-tls\") pod \"d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d\" (UID: \"d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d\") " Jul 07 16:19:18.100239 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:18.100032 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmz68\" (UniqueName: \"kubernetes.io/projected/d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d-kube-api-access-gmz68\") pod \"d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d\" (UID: \"d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d\") " Jul 07 16:19:18.100239 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:18.100062 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d-openshift-service-ca\") pod \"d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d\" (UID: \"d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d\") " Jul 07 16:19:18.100447 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:18.100421 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d" (UID: "d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:19:18.102251 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:18.102224 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d-kube-api-access-gmz68" (OuterVolumeSpecName: "kube-api-access-gmz68") pod "d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d" (UID: "d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d"). InnerVolumeSpecName "kube-api-access-gmz68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:19:18.102362 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:18.102258 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d" (UID: "d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:19:18.201678 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:18.201594 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d-maas-api-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:19:18.201678 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:18.201621 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmz68\" (UniqueName: \"kubernetes.io/projected/d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d-kube-api-access-gmz68\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:19:18.201678 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:18.201631 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d-openshift-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:19:18.212863 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:18.212824 2557 generic.go:356] "Generic (PLEG): container finished" podID="d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d" containerID="9185e1c28a38bff33865039bb9e81d64068f2136e97c2b7f1837dcf3b6e12133" exitCode=0 Jul 07 16:19:18.213042 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:18.212871 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" Jul 07 16:19:18.213042 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:18.212872 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" event={"ID":"d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d","Type":"ContainerDied","Data":"9185e1c28a38bff33865039bb9e81d64068f2136e97c2b7f1837dcf3b6e12133"} Jul 07 16:19:18.213042 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:18.213000 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh" event={"ID":"d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d","Type":"ContainerDied","Data":"b184227bd4ab42efd24107b947fd4eee9f17e435ad60d576454afb81e2959346"} Jul 07 16:19:18.213042 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:18.213028 2557 scope.go:117] "RemoveContainer" containerID="9185e1c28a38bff33865039bb9e81d64068f2136e97c2b7f1837dcf3b6e12133" Jul 07 16:19:18.223673 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:18.223652 2557 scope.go:117] "RemoveContainer" containerID="9185e1c28a38bff33865039bb9e81d64068f2136e97c2b7f1837dcf3b6e12133" Jul 07 16:19:18.224026 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:19:18.224006 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9185e1c28a38bff33865039bb9e81d64068f2136e97c2b7f1837dcf3b6e12133\": container with ID starting with 9185e1c28a38bff33865039bb9e81d64068f2136e97c2b7f1837dcf3b6e12133 not found: ID does not exist" containerID="9185e1c28a38bff33865039bb9e81d64068f2136e97c2b7f1837dcf3b6e12133" Jul 07 16:19:18.224103 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:18.224037 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9185e1c28a38bff33865039bb9e81d64068f2136e97c2b7f1837dcf3b6e12133"} err="failed to get container status \"9185e1c28a38bff33865039bb9e81d64068f2136e97c2b7f1837dcf3b6e12133\": rpc error: code = NotFound desc = could not find container \"9185e1c28a38bff33865039bb9e81d64068f2136e97c2b7f1837dcf3b6e12133\": container with ID starting with 9185e1c28a38bff33865039bb9e81d64068f2136e97c2b7f1837dcf3b6e12133 not found: ID does not exist" Jul 07 16:19:18.234424 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:18.234399 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh"] Jul 07 16:19:18.238659 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:18.238637 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-12f2df-5c67786857-c9rkh"] Jul 07 16:19:18.352194 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:18.352164 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4aa7bee-c3b4-4c91-815a-dd67e527684e" path="/var/lib/kubelet/pods/a4aa7bee-c3b4-4c91-815a-dd67e527684e/volumes" Jul 07 16:19:18.352615 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:19:18.352601 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d" path="/var/lib/kubelet/pods/d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d/volumes" Jul 07 16:20:00.693449 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:00.693412 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln"] Jul 07 16:20:00.694062 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:00.693647 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" podUID="0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8" containerName="maas-api" containerID="cri-o://9d9970966c72739b7efabbfca8bbfb63535abcba4c70881d6e4a9200934922d6" gracePeriod=30 Jul 07 16:20:00.732914 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:00.732874 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-ac6096-29724015-z4qvp"] Jul 07 16:20:00.736489 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:00.736464 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-ac6096-29724015-z4qvp"] Jul 07 16:20:00.948402 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:00.948339 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" Jul 07 16:20:01.109337 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:01.109298 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8-maas-api-tls\") pod \"0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8\" (UID: \"0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8\") " Jul 07 16:20:01.109515 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:01.109361 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8-openshift-service-ca\") pod \"0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8\" (UID: \"0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8\") " Jul 07 16:20:01.109515 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:01.109405 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whmvp\" (UniqueName: \"kubernetes.io/projected/0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8-kube-api-access-whmvp\") pod \"0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8\" (UID: \"0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8\") " Jul 07 16:20:01.109779 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:01.109745 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8" (UID: "0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:20:01.111568 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:01.111540 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8" (UID: "0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:20:01.111680 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:01.111547 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8-kube-api-access-whmvp" (OuterVolumeSpecName: "kube-api-access-whmvp") pod "0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8" (UID: "0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8"). InnerVolumeSpecName "kube-api-access-whmvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:20:01.210695 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:01.210600 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whmvp\" (UniqueName: \"kubernetes.io/projected/0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8-kube-api-access-whmvp\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:20:01.210695 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:01.210634 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8-maas-api-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:20:01.210695 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:01.210645 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8-openshift-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\"" Jul 07 16:20:01.484708 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:01.484614 2557 generic.go:356] "Generic (PLEG): container finished" podID="0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8" containerID="9d9970966c72739b7efabbfca8bbfb63535abcba4c70881d6e4a9200934922d6" exitCode=0 Jul 07 16:20:01.484708 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:01.484674 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" Jul 07 16:20:01.484708 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:01.484696 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" event={"ID":"0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8","Type":"ContainerDied","Data":"9d9970966c72739b7efabbfca8bbfb63535abcba4c70881d6e4a9200934922d6"} Jul 07 16:20:01.484976 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:01.484753 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln" event={"ID":"0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8","Type":"ContainerDied","Data":"4022ba13873b1aa4496c0df942ff82be2b53ae302e938b051a0ab325c043c6ea"} Jul 07 16:20:01.484976 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:01.484776 2557 scope.go:117] "RemoveContainer" containerID="9d9970966c72739b7efabbfca8bbfb63535abcba4c70881d6e4a9200934922d6" Jul 07 16:20:01.495431 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:01.495405 2557 scope.go:117] "RemoveContainer" containerID="9d9970966c72739b7efabbfca8bbfb63535abcba4c70881d6e4a9200934922d6" Jul 07 16:20:01.495701 ip-10-0-139-248 kubenswrapper[2557]: E0707 16:20:01.495685 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d9970966c72739b7efabbfca8bbfb63535abcba4c70881d6e4a9200934922d6\": container with ID starting with 9d9970966c72739b7efabbfca8bbfb63535abcba4c70881d6e4a9200934922d6 not found: ID does not exist" containerID="9d9970966c72739b7efabbfca8bbfb63535abcba4c70881d6e4a9200934922d6" Jul 07 16:20:01.495745 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:01.495710 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9970966c72739b7efabbfca8bbfb63535abcba4c70881d6e4a9200934922d6"} err="failed to get container status \"9d9970966c72739b7efabbfca8bbfb63535abcba4c70881d6e4a9200934922d6\": rpc error: code = NotFound desc = could not find container \"9d9970966c72739b7efabbfca8bbfb63535abcba4c70881d6e4a9200934922d6\": container with ID starting with 9d9970966c72739b7efabbfca8bbfb63535abcba4c70881d6e4a9200934922d6 not found: ID does not exist" Jul 07 16:20:01.508355 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:01.508323 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln"] Jul 07 16:20:01.512814 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:01.512789 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-ac6096-667b7c5cd8-94lln"] Jul 07 16:20:02.351299 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:02.351261 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8" path="/var/lib/kubelet/pods/0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8/volumes" Jul 07 16:20:02.351668 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:02.351615 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="890454b7-b54c-4796-b587-b3b8b9e0306c" path="/var/lib/kubelet/pods/890454b7-b54c-4796-b587-b3b8b9e0306c/volumes" Jul 07 16:20:12.107791 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:12.107757 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-6h895_34784301-08df-4c57-b141-07a592143554/manager/0.log" Jul 07 16:20:12.240131 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:12.240081 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-9qxdl_3ed630e1-4529-4df0-840a-748a59a09b1a/manager/0.log" Jul 07 16:20:12.387866 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:12.387790 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-55c8bccfbb-hh996_53121f11-e81a-4ed0-a610-78a82d907029/maas-api/0.log" Jul 07 16:20:12.503994 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:12.503959 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724000-p7lq6_7b16651f-b2cb-4e87-bc5a-3684c373e07a/cleanup/0.log" Jul 07 16:20:12.617559 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:12.617533 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724015-mtwsp_77b65a9e-fb4f-49c2-be4e-0c463535d64d/cleanup/0.log" Jul 07 16:20:12.746167 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:12.746065 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67dfc6764d-dbhsj_6ef0c63c-e464-461a-9933-bae62726907d/manager/0.log" Jul 07 16:20:12.874406 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:12.874378 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-d4cjs_69ad1c34-3a8c-4fdd-b2f0-6182f7bf62b3/server/0.log" Jul 07 16:20:13.114886 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:13.114855 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77db849c7c-4wnc8_3134dcbe-f349-45f2-9176-5cf7a2026c64/copy-manifests/0.log" Jul 07 16:20:13.123303 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:13.123279 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77db849c7c-4wnc8_3134dcbe-f349-45f2-9176-5cf7a2026c64/manager/0.log" Jul 07 16:20:14.274309 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:14.274272 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr_bae8fc14-9c5a-4ed3-b35a-0406be00443e/pull/0.log" Jul 07 16:20:14.280377 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:14.280348 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr_bae8fc14-9c5a-4ed3-b35a-0406be00443e/extract/0.log" Jul 07 16:20:14.285853 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:14.285833 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr_bae8fc14-9c5a-4ed3-b35a-0406be00443e/util/0.log" Jul 07 16:20:14.398410 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:14.398382 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9_026d55bd-3c74-427f-a8c9-62b3011eb34c/util/0.log" Jul 07 16:20:14.405308 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:14.405274 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9_026d55bd-3c74-427f-a8c9-62b3011eb34c/pull/0.log" Jul 07 16:20:14.412092 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:14.412072 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9_026d55bd-3c74-427f-a8c9-62b3011eb34c/extract/0.log" Jul 07 16:20:14.530857 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:14.530782 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc_f53c3f49-cca3-4660-96d2-a3a32b5a39c6/util/0.log" Jul 07 16:20:14.537795 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:14.537770 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc_f53c3f49-cca3-4660-96d2-a3a32b5a39c6/pull/0.log" Jul 07 16:20:14.544782 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:14.544753 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc_f53c3f49-cca3-4660-96d2-a3a32b5a39c6/extract/0.log" Jul 07 16:20:14.657575 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:14.657543 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4_faef017c-9227-481f-b21d-a8b7c9a214c4/extract/0.log" Jul 07 16:20:14.664015 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:14.663990 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4_faef017c-9227-481f-b21d-a8b7c9a214c4/util/0.log" Jul 07 16:20:14.670415 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:14.670394 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4_faef017c-9227-481f-b21d-a8b7c9a214c4/pull/0.log" Jul 07 16:20:14.901442 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:14.901406 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-mjfpk_5de713ba-3082-4e16-9015-28a02e8dcb8c/manager/0.log" Jul 07 16:20:15.016698 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:15.016664 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-b5fhk_b3c22ee8-1a94-4949-8438-91a59f55ef04/manager/0.log" Jul 07 16:20:15.127128 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:15.127090 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-qxpkg_4184c3b8-50a2-4d79-8afa-0eefe6132943/kuadrant-console-plugin/0.log" Jul 07 16:20:15.250633 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:15.250550 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-6lgzr_ec46ef29-6d38-44fd-a742-e748bec0ffe6/registry-server/0.log" Jul 07 16:20:15.376322 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:15.376289 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-bgcld_670b1e18-e596-4558-8bba-123a1f17d805/manager/0.log" Jul 07 16:20:15.620881 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:15.620852 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-jcw8t_6d816f57-931a-4919-91c6-fec98b115825/manager/0.log" Jul 07 16:20:16.000687 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:16.000611 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4smx5t_7607df7a-3a72-46ef-b035-74355b31932a/istio-proxy/0.log" Jul 07 16:20:16.332859 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:16.332830 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6669df4868-vm6h8_00e096f9-6fb3-4444-ae6b-60daf719f0de/kube-auth-proxy/0.log" Jul 07 16:20:16.542475 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:16.542445 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/1.log" Jul 07 16:20:16.547404 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:16.547384 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hkj2z_07a80436-65b9-4f0d-99b3-ecda36240b48/ovn-acl-logging/0.log" Jul 07 16:20:16.550431 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:16.550412 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/1.log" Jul 07 16:20:16.554799 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:16.554782 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hkj2z_07a80436-65b9-4f0d-99b3-ecda36240b48/ovn-acl-logging/0.log" Jul 07 16:20:16.723324 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:16.723290 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-xplfd_ceeafdcc-0cce-4dcb-828c-93b0965a4b54/payload-processing/0.log" Jul 07 16:20:16.831829 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:16.831787 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b5cfc467-zp2s8_6937108d-00d9-456e-80a3-5230106a0708/router/0.log" Jul 07 16:20:17.418958 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:17.418931 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-bhvw8_dd5cc899-ae05-435a-8aa4-4d0044000701/main/0.log" Jul 07 16:20:17.548292 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:17.548262 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvztw9_c435d3d2-3245-44ba-8f34-fd99c41e8e67/main/0.log" Jul 07 16:20:25.060124 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:25.060082 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-q2m8b_b99c24f7-0333-47fe-8b1a-ed9c6a10d23c/global-pull-secret-syncer/0.log" Jul 07 16:20:25.159997 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:25.159966 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vktnc_3567278c-a1c9-403d-9353-80da294fafb9/konnectivity-agent/0.log" Jul 07 16:20:25.270758 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:25.270726 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-248.ec2.internal_d0a6e48ccbac6ae57118c8e5af923f7b/haproxy/0.log" Jul 07 16:20:29.911788 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:29.911754 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr_bae8fc14-9c5a-4ed3-b35a-0406be00443e/extract/0.log" Jul 07 16:20:29.938087 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:29.938057 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr_bae8fc14-9c5a-4ed3-b35a-0406be00443e/util/0.log" Jul 07 16:20:29.964080 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:29.964050 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sjkfr_bae8fc14-9c5a-4ed3-b35a-0406be00443e/pull/0.log" Jul 07 16:20:29.998098 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:29.998070 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9_026d55bd-3c74-427f-a8c9-62b3011eb34c/extract/0.log" Jul 07 16:20:30.029620 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:30.029593 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9_026d55bd-3c74-427f-a8c9-62b3011eb34c/util/0.log" Jul 07 16:20:30.059910 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:30.059880 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0467s9_026d55bd-3c74-427f-a8c9-62b3011eb34c/pull/0.log" Jul 07 16:20:30.097750 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:30.097722 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc_f53c3f49-cca3-4660-96d2-a3a32b5a39c6/extract/0.log" Jul 07 16:20:30.126024 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:30.125990 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc_f53c3f49-cca3-4660-96d2-a3a32b5a39c6/util/0.log" Jul 07 16:20:30.160513 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:30.160479 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dvfmc_f53c3f49-cca3-4660-96d2-a3a32b5a39c6/pull/0.log" Jul 07 16:20:30.190550 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:30.190471 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4_faef017c-9227-481f-b21d-a8b7c9a214c4/extract/0.log" Jul 07 16:20:30.218246 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:30.218218 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4_faef017c-9227-481f-b21d-a8b7c9a214c4/util/0.log" Jul 07 16:20:30.243207 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:30.243175 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gbxq4_faef017c-9227-481f-b21d-a8b7c9a214c4/pull/0.log" Jul 07 16:20:30.666807 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:30.666775 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-mjfpk_5de713ba-3082-4e16-9015-28a02e8dcb8c/manager/0.log" Jul 07 16:20:30.693725 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:30.693690 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-b5fhk_b3c22ee8-1a94-4949-8438-91a59f55ef04/manager/0.log" Jul 07 16:20:30.722466 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:30.722435 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-qxpkg_4184c3b8-50a2-4d79-8afa-0eefe6132943/kuadrant-console-plugin/0.log" Jul 07 16:20:30.778721 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:30.778689 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-6lgzr_ec46ef29-6d38-44fd-a742-e748bec0ffe6/registry-server/0.log" Jul 07 16:20:30.935744 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:30.935651 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-bgcld_670b1e18-e596-4558-8bba-123a1f17d805/manager/0.log" Jul 07 16:20:31.076760 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:31.076702 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-jcw8t_6d816f57-931a-4919-91c6-fec98b115825/manager/0.log" Jul 07 16:20:32.833962 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:32.833926 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-8mf6j_cdd40d24-e30b-496c-b92b-7c80e6ee0b33/cluster-monitoring-operator/0.log" Jul 07 16:20:33.304301 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:33.304264 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vvrpg_5c089f7e-a5c2-4343-8074-d016179669ae/node-exporter/0.log" Jul 07 16:20:33.329372 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:33.329344 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vvrpg_5c089f7e-a5c2-4343-8074-d016179669ae/kube-rbac-proxy/0.log" Jul 07 16:20:33.355776 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:33.355752 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vvrpg_5c089f7e-a5c2-4343-8074-d016179669ae/init-textfile/0.log" Jul 07 16:20:33.713155 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:33.713047 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-j7997_8518e838-f15a-4322-b246-5a439d4c6b62/prometheus-operator-admission-webhook/0.log" Jul 07 16:20:34.679864 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.679823 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf"] Jul 07 16:20:34.680302 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.680250 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4aa7bee-c3b4-4c91-815a-dd67e527684e" containerName="cleanup" Jul 07 16:20:34.680302 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.680261 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4aa7bee-c3b4-4c91-815a-dd67e527684e" containerName="cleanup" Jul 07 16:20:34.680302 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.680275 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="890454b7-b54c-4796-b587-b3b8b9e0306c" containerName="cleanup" Jul 07 16:20:34.680302 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.680281 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="890454b7-b54c-4796-b587-b3b8b9e0306c" containerName="cleanup" Jul 07 16:20:34.680302 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.680292 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77b65a9e-fb4f-49c2-be4e-0c463535d64d" containerName="cleanup" Jul 07 16:20:34.680302 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.680297 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="77b65a9e-fb4f-49c2-be4e-0c463535d64d" containerName="cleanup" Jul 07 16:20:34.680517 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.680316 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d" containerName="maas-api" Jul 07 16:20:34.680517 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.680321 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d" containerName="maas-api" Jul 07 16:20:34.680517 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.680330 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8" containerName="maas-api" Jul 07 16:20:34.680517 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.680335 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8" containerName="maas-api" Jul 07 16:20:34.680517 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.680391 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0512faf5-78bb-4ba1-bdd7-0bf1cb4ff2d8" containerName="maas-api" Jul 07 16:20:34.680517 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.680398 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1ad92e6-1a34-4b36-bd2b-a507e8ebe50d" containerName="maas-api" Jul 07 16:20:34.680517 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.680404 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="890454b7-b54c-4796-b587-b3b8b9e0306c" containerName="cleanup" Jul 07 16:20:34.680517 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.680412 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4aa7bee-c3b4-4c91-815a-dd67e527684e" containerName="cleanup" Jul 07 16:20:34.680517 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.680419 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="77b65a9e-fb4f-49c2-be4e-0c463535d64d" containerName="cleanup" Jul 07 16:20:34.683832 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.683816 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf" Jul 07 16:20:34.685896 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.685861 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lhshj\"/\"openshift-service-ca.crt\"" Jul 07 16:20:34.686016 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.685952 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lhshj\"/\"kube-root-ca.crt\"" Jul 07 16:20:34.686016 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.686008 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lhshj\"/\"default-dockercfg-wfbq2\"" Jul 07 16:20:34.694265 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.694235 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf"] Jul 07 16:20:34.793432 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.793395 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d01aeffa-3be1-4d94-b3b4-942e34b4c802-lib-modules\") pod \"perf-node-gather-daemonset-6wzjf\" (UID: \"d01aeffa-3be1-4d94-b3b4-942e34b4c802\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf" Jul 07 16:20:34.793626 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.793445 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d01aeffa-3be1-4d94-b3b4-942e34b4c802-podres\") pod \"perf-node-gather-daemonset-6wzjf\" (UID: \"d01aeffa-3be1-4d94-b3b4-942e34b4c802\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf" Jul 07 16:20:34.793626 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.793472 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zdbh\" (UniqueName: \"kubernetes.io/projected/d01aeffa-3be1-4d94-b3b4-942e34b4c802-kube-api-access-2zdbh\") pod \"perf-node-gather-daemonset-6wzjf\" (UID: \"d01aeffa-3be1-4d94-b3b4-942e34b4c802\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf" Jul 07 16:20:34.793626 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.793542 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d01aeffa-3be1-4d94-b3b4-942e34b4c802-proc\") pod \"perf-node-gather-daemonset-6wzjf\" (UID: \"d01aeffa-3be1-4d94-b3b4-942e34b4c802\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf" Jul 07 16:20:34.793626 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.793575 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d01aeffa-3be1-4d94-b3b4-942e34b4c802-sys\") pod \"perf-node-gather-daemonset-6wzjf\" (UID: \"d01aeffa-3be1-4d94-b3b4-942e34b4c802\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf" Jul 07 16:20:34.894345 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.894304 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d01aeffa-3be1-4d94-b3b4-942e34b4c802-lib-modules\") pod \"perf-node-gather-daemonset-6wzjf\" (UID: \"d01aeffa-3be1-4d94-b3b4-942e34b4c802\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf" Jul 07 16:20:34.894345 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.894348 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d01aeffa-3be1-4d94-b3b4-942e34b4c802-podres\") pod \"perf-node-gather-daemonset-6wzjf\" (UID: \"d01aeffa-3be1-4d94-b3b4-942e34b4c802\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf" Jul 07 16:20:34.894609 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.894386 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zdbh\" (UniqueName: \"kubernetes.io/projected/d01aeffa-3be1-4d94-b3b4-942e34b4c802-kube-api-access-2zdbh\") pod \"perf-node-gather-daemonset-6wzjf\" (UID: \"d01aeffa-3be1-4d94-b3b4-942e34b4c802\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf" Jul 07 16:20:34.894609 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.894443 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d01aeffa-3be1-4d94-b3b4-942e34b4c802-proc\") pod \"perf-node-gather-daemonset-6wzjf\" (UID: \"d01aeffa-3be1-4d94-b3b4-942e34b4c802\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf" Jul 07 16:20:34.894609 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.894474 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d01aeffa-3be1-4d94-b3b4-942e34b4c802-sys\") pod \"perf-node-gather-daemonset-6wzjf\" (UID: \"d01aeffa-3be1-4d94-b3b4-942e34b4c802\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf" Jul 07 16:20:34.894609 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.894486 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d01aeffa-3be1-4d94-b3b4-942e34b4c802-lib-modules\") pod \"perf-node-gather-daemonset-6wzjf\" (UID: \"d01aeffa-3be1-4d94-b3b4-942e34b4c802\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf" Jul 07 16:20:34.894609 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.894535 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d01aeffa-3be1-4d94-b3b4-942e34b4c802-proc\") pod \"perf-node-gather-daemonset-6wzjf\" (UID: \"d01aeffa-3be1-4d94-b3b4-942e34b4c802\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf" Jul 07 16:20:34.894806 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.894615 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d01aeffa-3be1-4d94-b3b4-942e34b4c802-sys\") pod \"perf-node-gather-daemonset-6wzjf\" (UID: \"d01aeffa-3be1-4d94-b3b4-942e34b4c802\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf" Jul 07 16:20:34.894806 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.894668 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d01aeffa-3be1-4d94-b3b4-942e34b4c802-podres\") pod \"perf-node-gather-daemonset-6wzjf\" (UID: \"d01aeffa-3be1-4d94-b3b4-942e34b4c802\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf" Jul 07 16:20:34.902896 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.902864 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zdbh\" (UniqueName: \"kubernetes.io/projected/d01aeffa-3be1-4d94-b3b4-942e34b4c802-kube-api-access-2zdbh\") pod \"perf-node-gather-daemonset-6wzjf\" (UID: \"d01aeffa-3be1-4d94-b3b4-942e34b4c802\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf" Jul 07 16:20:34.995368 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:34.995277 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf" Jul 07 16:20:35.040949 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:35.040921 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-vfwk5_a278ee42-38b2-4aee-b359-602e3bbf59be/networking-console-plugin/0.log" Jul 07 16:20:35.135673 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:35.135641 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf"] Jul 07 16:20:35.137215 ip-10-0-139-248 kubenswrapper[2557]: W0707 16:20:35.137184 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd01aeffa_3be1_4d94_b3b4_942e34b4c802.slice/crio-975980cd5b4ba44d806f3cb17c884da8513fc050c4e7a38d346d991d22c87fec WatchSource:0}: Error finding container 975980cd5b4ba44d806f3cb17c884da8513fc050c4e7a38d346d991d22c87fec: Status 404 returned error can't find the container with id 975980cd5b4ba44d806f3cb17c884da8513fc050c4e7a38d346d991d22c87fec Jul 07 16:20:35.697833 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:35.697802 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/1.log" Jul 07 16:20:35.702527 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:35.702482 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf" event={"ID":"d01aeffa-3be1-4d94-b3b4-942e34b4c802","Type":"ContainerStarted","Data":"57be5aa3a1d1d1ce03c60367d3b22985e842e6d28cf8615c9286ccc39156c0bd"} Jul 07 16:20:35.702727 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:35.702549 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf" event={"ID":"d01aeffa-3be1-4d94-b3b4-942e34b4c802","Type":"ContainerStarted","Data":"975980cd5b4ba44d806f3cb17c884da8513fc050c4e7a38d346d991d22c87fec"} Jul 07 16:20:35.702727 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:35.702609 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf" Jul 07 16:20:35.706731 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:35.706710 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-dn2nk_d1fdbe6c-661b-4201-87f1-7ca6fe2b561b/console-operator/2.log" Jul 07 16:20:35.720320 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:35.720269 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf" podStartSLOduration=1.7202547099999999 podStartE2EDuration="1.72025471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:20:35.718356013 +0000 UTC m=+2419.896000465" watchObservedRunningTime="2026-07-07 16:20:35.72025471 +0000 UTC m=+2419.897899159" Jul 07 16:20:36.216297 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:36.216267 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-677459dbf6-zglpq_34dc6e32-5545-420f-a4cd-f96c249169fa/console/0.log" Jul 07 16:20:36.257290 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:36.257262 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-6zs7q_e26092ce-3f22-4a9e-b816-428098ee2569/download-server/0.log" Jul 07 16:20:36.834837 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:36.834785 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-vsnfw_55083ac2-3c7e-4664-995d-4d0ab74e3a20/volume-data-source-validator/0.log" Jul 07 16:20:37.913732 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:37.913699 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-spwqp_4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3/dns/0.log" Jul 07 16:20:37.937075 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:37.937040 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-spwqp_4eda6cd2-bb41-4f2e-8c1e-1447a69b29a3/kube-rbac-proxy/0.log" Jul 07 16:20:38.043367 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:38.043340 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lhvjn_e7b29c32-fe48-437a-956a-f4b0d4eab6c3/dns-node-resolver/0.log" Jul 07 16:20:38.693937 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:38.693907 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mcrlt_78ecda82-54b3-4727-89b9-5d55dd0ca428/node-ca/0.log" Jul 07 16:20:39.644370 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:39.644335 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4smx5t_7607df7a-3a72-46ef-b035-74355b31932a/istio-proxy/0.log" Jul 07 16:20:39.889016 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:39.888981 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6669df4868-vm6h8_00e096f9-6fb3-4444-ae6b-60daf719f0de/kube-auth-proxy/0.log" Jul 07 16:20:40.097081 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:40.097042 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-xplfd_ceeafdcc-0cce-4dcb-828c-93b0965a4b54/payload-processing/0.log" Jul 07 16:20:40.122701 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:40.122676 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b5cfc467-zp2s8_6937108d-00d9-456e-80a3-5230106a0708/router/0.log" Jul 07 16:20:40.850025 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:40.849985 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rk7b5_a9aad135-443d-43c6-8b9d-94107c57ef21/serve-healthcheck-canary/0.log" Jul 07 16:20:41.356429 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:41.356392 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-v6x9c_37e8dea9-9c9a-41f5-bd64-6d0331ced375/insights-operator/0.log" Jul 07 16:20:41.357619 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:41.357596 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-v6x9c_37e8dea9-9c9a-41f5-bd64-6d0331ced375/insights-operator/1.log" Jul 07 16:20:41.535621 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:41.535591 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q4x4h_0eb10328-3df9-435e-91e4-02e0a6731581/kube-rbac-proxy/0.log" Jul 07 16:20:41.559099 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:41.559072 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q4x4h_0eb10328-3df9-435e-91e4-02e0a6731581/exporter/0.log" Jul 07 16:20:41.583381 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:41.583346 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q4x4h_0eb10328-3df9-435e-91e4-02e0a6731581/extractor/0.log" Jul 07 16:20:41.719833 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:41.719755 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-6wzjf" Jul 07 16:20:43.859972 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:43.859893 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-6h895_34784301-08df-4c57-b141-07a592143554/manager/0.log" Jul 07 16:20:44.108438 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:44.108397 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-9qxdl_3ed630e1-4529-4df0-840a-748a59a09b1a/manager/0.log" Jul 07 16:20:44.162243 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:44.162160 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-55c8bccfbb-hh996_53121f11-e81a-4ed0-a610-78a82d907029/maas-api/0.log" Jul 07 16:20:44.195765 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:44.195736 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724000-p7lq6_7b16651f-b2cb-4e87-bc5a-3684c373e07a/cleanup/0.log" Jul 07 16:20:44.236496 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:44.236473 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724015-mtwsp_77b65a9e-fb4f-49c2-be4e-0c463535d64d/cleanup/0.log" Jul 07 16:20:44.407887 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:44.407852 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67dfc6764d-dbhsj_6ef0c63c-e464-461a-9933-bae62726907d/manager/0.log" Jul 07 16:20:44.445171 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:44.445057 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-d4cjs_69ad1c34-3a8c-4fdd-b2f0-6182f7bf62b3/server/0.log" Jul 07 16:20:44.585685 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:44.585638 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77db849c7c-4wnc8_3134dcbe-f349-45f2-9176-5cf7a2026c64/manager/0.log" Jul 07 16:20:44.622506 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:44.622483 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77db849c7c-4wnc8_3134dcbe-f349-45f2-9176-5cf7a2026c64/copy-manifests/0.log" Jul 07 16:20:45.655976 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:45.655944 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4smx5t_7607df7a-3a72-46ef-b035-74355b31932a/istio-proxy/0.log" Jul 07 16:20:45.880920 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:45.880877 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6669df4868-vm6h8_00e096f9-6fb3-4444-ae6b-60daf719f0de/kube-auth-proxy/0.log" Jul 07 16:20:46.059625 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:46.059592 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-xplfd_ceeafdcc-0cce-4dcb-828c-93b0965a4b54/payload-processing/0.log" Jul 07 16:20:46.081183 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:46.081157 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b5cfc467-zp2s8_6937108d-00d9-456e-80a3-5230106a0708/router/0.log" Jul 07 16:20:46.186428 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:46.186395 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-hwnl8_a554767e-512e-4418-88cf-49b2cf7f3e79/openshift-lws-operator/0.log" Jul 07 16:20:47.254077 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:47.254043 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4smx5t_7607df7a-3a72-46ef-b035-74355b31932a/istio-proxy/0.log" Jul 07 16:20:47.476931 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:47.476900 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6669df4868-vm6h8_00e096f9-6fb3-4444-ae6b-60daf719f0de/kube-auth-proxy/0.log" Jul 07 16:20:47.663282 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:47.663251 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-xplfd_ceeafdcc-0cce-4dcb-828c-93b0965a4b54/payload-processing/0.log" Jul 07 16:20:47.681362 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:47.681337 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b5cfc467-zp2s8_6937108d-00d9-456e-80a3-5230106a0708/router/0.log" Jul 07 16:20:51.071179 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:51.071149 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-p487q_8148a32a-727e-4d27-9cc5-43b0b3a8f89c/migrator/0.log" Jul 07 16:20:51.096032 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:51.096003 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-p487q_8148a32a-727e-4d27-9cc5-43b0b3a8f89c/graceful-termination/0.log" Jul 07 16:20:51.492081 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:51.491996 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-kt6w8_62417042-0998-4d6a-ba6a-71ec72bde8e6/kube-storage-version-migrator-operator/1.log" Jul 07 16:20:51.494226 ip-10-0-139-248 kubenswrapper[2557]: I0707 16:20:51.494192 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-kt6w8_62417042-0998-4d6a-ba6a-71ec72bde8e6/kube-storage-version-migrator-operator/0.log"