Jul 06 21:12:53.292682 ip-10-0-140-54 systemd[1]: Starting Kubernetes Kubelet... Jul 06 21:12:53.880632 ip-10-0-140-54 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 21:12:53.880632 ip-10-0-140-54 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 21:12:53.880632 ip-10-0-140-54 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 21:12:53.880632 ip-10-0-140-54 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 21:12:53.880632 ip-10-0-140-54 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 21:12:53.880632 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.715016 2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 21:12:53.880632 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.719934 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 21:12:53.880632 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.719955 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 21:12:53.880632 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.719959 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 21:12:53.880632 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.719962 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 21:12:53.880632 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.719965 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 21:12:53.880632 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.719968 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 21:12:53.880632 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.719971 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 21:12:53.880632 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.719974 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 21:12:53.794118 ip-10-0-140-54 systemd[1]: Started Kubernetes Kubelet. Jul 06 21:12:54.033165 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.719976 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 21:12:54.033165 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.719979 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 21:12:54.033165 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.719982 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 21:12:54.033165 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.719984 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 21:12:54.033165 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.719988 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 21:12:54.033165 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.719991 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 21:12:54.033165 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.719994 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 21:12:54.033165 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.719996 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 21:12:54.033165 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.719999 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 21:12:54.033165 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720001 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 21:12:54.033165 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720004 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 21:12:54.033165 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720012 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 21:12:54.033165 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720016 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 21:12:54.033165 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720018 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 21:12:54.033165 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720021 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 06 21:12:54.033165 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720023 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 21:12:54.033165 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720027 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 21:12:54.033165 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720030 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 21:12:54.033165 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720032 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 21:12:54.033165 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720035 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 21:12:54.034446 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720037 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 21:12:54.034446 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720040 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 21:12:54.034446 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720043 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 21:12:54.034446 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720045 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 21:12:54.034446 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720048 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 21:12:54.034446 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720051 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 21:12:54.034446 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720053 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 21:12:54.034446 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720057 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 21:12:54.034446 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720060 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 21:12:54.034446 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720062 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 21:12:54.034446 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720065 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 21:12:54.034446 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720067 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 21:12:54.034446 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720070 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 21:12:54.034446 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720072 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 21:12:54.034446 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720075 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 21:12:54.034446 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720077 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 21:12:54.034446 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720080 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 21:12:54.034446 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720082 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 21:12:54.034446 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720085 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 21:12:54.035367 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720087 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 21:12:54.035367 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720091 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 21:12:54.035367 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720094 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 21:12:54.035367 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720096 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 21:12:54.035367 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720099 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 21:12:54.035367 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720101 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 21:12:54.035367 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720104 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 21:12:54.035367 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720106 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 21:12:54.035367 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720108 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 21:12:54.035367 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720111 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 21:12:54.035367 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720114 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 21:12:54.035367 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720117 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 21:12:54.035367 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720121 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 21:12:54.035367 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720123 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 21:12:54.035367 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720126 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 21:12:54.035367 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720129 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 21:12:54.035367 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720134 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 21:12:54.035367 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720140 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 21:12:54.035367 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720143 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 21:12:54.035367 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720146 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 21:12:54.035983 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720149 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 21:12:54.035983 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720151 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 21:12:54.035983 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720154 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 21:12:54.035983 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720158 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 21:12:54.035983 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720162 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 21:12:54.035983 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720164 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 21:12:54.035983 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720167 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 21:12:54.035983 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720169 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 21:12:54.035983 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720172 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 21:12:54.035983 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720174 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 21:12:54.035983 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720177 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 21:12:54.035983 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720181 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 21:12:54.035983 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720185 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 21:12:54.035983 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720189 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 21:12:54.035983 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720192 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 21:12:54.035983 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720196 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 21:12:54.035983 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720199 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 21:12:54.035983 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720202 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 21:12:54.035983 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720205 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 21:12:54.035983 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720207 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 21:12:54.036618 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720210 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 21:12:54.036618 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720213 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 21:12:54.036618 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720216 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 21:12:54.036618 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720218 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 21:12:54.036618 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720221 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 21:12:54.036618 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720224 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 21:12:54.036618 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720226 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 21:12:54.036618 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720229 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 21:12:54.036618 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720231 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 21:12:54.036618 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720234 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 21:12:54.036618 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720236 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 21:12:54.036618 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720239 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 21:12:54.036618 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720984 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 21:12:54.036618 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720992 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 21:12:54.036618 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720995 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 21:12:54.036618 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.720998 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 21:12:54.036618 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721000 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 21:12:54.036618 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721003 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 21:12:54.036618 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721005 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 21:12:54.036618 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721008 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 21:12:54.037185 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721011 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 21:12:54.037185 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721013 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 21:12:54.037185 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721016 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 21:12:54.037185 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721019 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 21:12:54.037185 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721021 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 21:12:54.037185 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721023 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 21:12:54.037185 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721026 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 21:12:54.037185 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721029 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 21:12:54.037185 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721032 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 21:12:54.037185 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721034 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 21:12:54.037185 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721037 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 21:12:54.037185 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721039 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 21:12:54.037185 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721041 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 21:12:54.037185 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721044 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 21:12:54.037185 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721046 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 21:12:54.037185 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721049 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 21:12:54.037185 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721051 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 21:12:54.037185 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721054 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 21:12:54.037185 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721057 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 21:12:54.037185 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721060 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 21:12:54.037840 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721062 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 21:12:54.037840 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721065 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 21:12:54.037840 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721068 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 21:12:54.037840 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721070 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 21:12:54.037840 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721073 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 21:12:54.037840 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721075 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 21:12:54.037840 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721078 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 21:12:54.037840 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721080 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 21:12:54.037840 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721082 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 06 21:12:54.037840 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721085 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 21:12:54.037840 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721087 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 21:12:54.037840 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721090 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 21:12:54.037840 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721092 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 21:12:54.037840 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721096 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 21:12:54.037840 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721100 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 21:12:54.037840 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721103 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 21:12:54.037840 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721105 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 21:12:54.037840 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721109 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 21:12:54.037840 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721111 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 21:12:54.038448 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721114 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 21:12:54.038448 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721120 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 21:12:54.038448 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721123 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 21:12:54.038448 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721126 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 21:12:54.038448 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721128 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 21:12:54.038448 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721130 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 21:12:54.038448 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721133 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 21:12:54.038448 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721135 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 21:12:54.038448 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721138 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 21:12:54.038448 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721140 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 21:12:54.038448 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721143 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 21:12:54.038448 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721146 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 21:12:54.038448 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721149 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 21:12:54.038448 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721151 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 21:12:54.038448 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721154 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 21:12:54.038448 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721156 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 21:12:54.038448 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721159 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 21:12:54.038448 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721164 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 21:12:54.038448 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721167 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 21:12:54.039037 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721171 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 21:12:54.039037 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721174 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 21:12:54.039037 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721176 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 21:12:54.039037 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721179 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 21:12:54.039037 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721181 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 21:12:54.039037 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721184 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 21:12:54.039037 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721186 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 21:12:54.039037 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721189 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 21:12:54.039037 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721191 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 21:12:54.039037 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721194 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 21:12:54.039037 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721196 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 21:12:54.039037 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721199 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 21:12:54.039037 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721201 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 21:12:54.039037 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721204 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 21:12:54.039037 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721208 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 21:12:54.039037 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721212 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 21:12:54.039037 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721215 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 21:12:54.039037 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721218 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 21:12:54.039037 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721221 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 21:12:54.039037 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721223 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 21:12:54.039629 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721226 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 21:12:54.039629 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721229 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 21:12:54.039629 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721232 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 21:12:54.039629 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721235 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 21:12:54.039629 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721237 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 21:12:54.039629 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721240 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 21:12:54.039629 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721244 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 21:12:54.039629 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721246 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 21:12:54.039629 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721250 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 21:12:54.039629 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721253 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 21:12:54.039629 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721258 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 21:12:54.039629 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721261 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 21:12:54.039629 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721268 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 21:12:54.039629 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721675 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 21:12:54.039629 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721681 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 21:12:54.039629 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721684 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 06 21:12:54.039629 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721687 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 21:12:54.039629 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721691 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 21:12:54.039629 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721693 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 21:12:54.040223 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721696 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 21:12:54.040223 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721699 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 21:12:54.040223 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721701 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 21:12:54.040223 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721704 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 21:12:54.040223 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721707 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 21:12:54.040223 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721709 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 21:12:54.040223 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721712 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 21:12:54.040223 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721714 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 21:12:54.040223 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721717 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 21:12:54.040223 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721720 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 21:12:54.040223 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721722 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 21:12:54.040223 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721725 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 21:12:54.040223 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721727 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 21:12:54.040223 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721731 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 21:12:54.040223 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721735 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 21:12:54.040223 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721737 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 21:12:54.040223 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721740 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 21:12:54.040223 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721743 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 21:12:54.040223 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721745 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 21:12:54.040223 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721748 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 21:12:54.040809 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721751 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 21:12:54.040809 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721753 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 21:12:54.040809 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721756 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 21:12:54.040809 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721761 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 21:12:54.040809 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721763 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 21:12:54.040809 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721767 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 21:12:54.040809 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721770 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 21:12:54.040809 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721772 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 21:12:54.040809 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721775 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 21:12:54.040809 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721778 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 21:12:54.040809 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721780 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 21:12:54.040809 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721783 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 21:12:54.040809 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721785 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 21:12:54.040809 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721787 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 21:12:54.040809 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721790 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 21:12:54.040809 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721793 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 21:12:54.040809 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721795 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 21:12:54.040809 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721798 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 21:12:54.040809 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721801 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 21:12:54.040809 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721804 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 21:12:54.041434 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721806 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 21:12:54.041434 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721809 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 21:12:54.041434 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721811 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 21:12:54.041434 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721814 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 21:12:54.041434 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721816 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 21:12:54.041434 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721818 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 21:12:54.041434 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721821 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 21:12:54.041434 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721823 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 21:12:54.041434 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721826 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 21:12:54.041434 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721828 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 21:12:54.041434 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721831 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 21:12:54.041434 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721833 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 21:12:54.041434 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721836 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 21:12:54.041434 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721838 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 21:12:54.041434 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721841 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 21:12:54.041434 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721844 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 21:12:54.041434 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721847 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 21:12:54.041434 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721850 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 21:12:54.041434 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721853 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 21:12:54.041434 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721855 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 21:12:54.041984 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721858 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 21:12:54.041984 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721860 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 21:12:54.041984 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721863 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 21:12:54.041984 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721866 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 21:12:54.041984 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721868 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 21:12:54.041984 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721871 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 21:12:54.041984 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721873 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 21:12:54.041984 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721876 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 21:12:54.041984 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721878 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 21:12:54.041984 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721881 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 21:12:54.041984 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721883 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 21:12:54.041984 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721886 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 21:12:54.041984 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721888 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 21:12:54.041984 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721892 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 21:12:54.041984 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721895 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 21:12:54.041984 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721897 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 21:12:54.041984 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721900 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 21:12:54.041984 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721902 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 21:12:54.041984 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721905 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 21:12:54.041984 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721907 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 21:12:54.042650 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721910 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 21:12:54.042650 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721912 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 21:12:54.042650 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721914 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 21:12:54.042650 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721917 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 21:12:54.042650 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721919 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 21:12:54.042650 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721922 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 21:12:54.042650 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721924 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 21:12:54.042650 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721927 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 21:12:54.042650 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721930 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 21:12:54.042650 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721934 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 21:12:54.042650 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721938 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 21:12:54.042650 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721941 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 21:12:54.042650 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.721944 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 21:12:54.042650 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722018 2581 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 21:12:54.042650 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722026 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 21:12:54.042650 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722033 2581 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 21:12:54.042650 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722037 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 21:12:54.042650 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722041 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 21:12:54.042650 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722045 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 21:12:54.042650 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722049 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722053 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722057 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722060 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722064 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722067 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722070 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722074 2581 flags.go:64] FLAG: --cgroup-root="" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722077 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722080 2581 flags.go:64] FLAG: --client-ca-file="" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722083 2581 flags.go:64] FLAG: --cloud-provider="external" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722086 2581 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722091 2581 flags.go:64] FLAG: --cluster-domain="" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722094 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722097 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722100 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722103 2581 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722107 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722110 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722113 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722118 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722121 2581 flags.go:64] FLAG: --contention-profiling="false" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722125 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722128 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 21:12:54.043186 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722131 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722134 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722138 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722141 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722144 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722147 2581 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722150 2581 flags.go:64] FLAG: --enable-server="true" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722153 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722158 2581 flags.go:64] FLAG: --event-burst="100" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722161 2581 flags.go:64] FLAG: --event-qps="50" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722163 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722167 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722170 2581 flags.go:64] FLAG: --eviction-hard="" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722174 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722176 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722179 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722182 2581 flags.go:64] FLAG: --eviction-soft="" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722186 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722189 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722192 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722195 2581 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722198 2581 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722200 2581 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722203 2581 flags.go:64] FLAG: --feature-gates="" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722207 2581 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 21:12:54.043854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722210 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722213 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722216 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722220 2581 flags.go:64] FLAG: --healthz-port="10248" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722223 2581 flags.go:64] FLAG: --help="false" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722227 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-140-54.ec2.internal" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722230 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722233 2581 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722236 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722240 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722244 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722247 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722250 2581 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722252 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722255 2581 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722258 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722261 2581 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722264 2581 flags.go:64] FLAG: --kube-reserved="" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722267 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722270 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722273 2581 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722275 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722278 2581 flags.go:64] FLAG: --lock-file="" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722281 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 21:12:54.044554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722284 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722288 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722294 2581 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722297 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722299 2581 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722302 2581 flags.go:64] FLAG: --logging-format="text" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722305 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722308 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722311 2581 flags.go:64] FLAG: --manifest-url="" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722314 2581 flags.go:64] FLAG: --manifest-url-header="" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722319 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722323 2581 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722327 2581 flags.go:64] FLAG: --max-pods="110" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722331 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722334 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722336 2581 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722340 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722343 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722346 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722349 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722356 2581 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722359 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722362 2581 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 21:12:54.045211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722365 2581 flags.go:64] FLAG: --pod-cidr="" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722368 2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722373 2581 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722376 2581 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722391 2581 flags.go:64] FLAG: --pods-per-core="0" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722394 2581 flags.go:64] FLAG: --port="10250" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722397 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722400 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cd15b2519c1d37de" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722403 2581 flags.go:64] FLAG: --qos-reserved="" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722407 2581 flags.go:64] FLAG: --read-only-port="10255" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722410 2581 flags.go:64] FLAG: --register-node="true" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722413 2581 flags.go:64] FLAG: --register-with-taints="" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722417 2581 flags.go:64] FLAG: --registry-burst="10" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722420 2581 flags.go:64] FLAG: --registry-qps="5" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722423 2581 flags.go:64] FLAG: --reserved-cpus="" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722426 2581 flags.go:64] FLAG: --reserved-memory="" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722429 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722432 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722435 2581 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722438 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722441 2581 flags.go:64] FLAG: --runonce="false" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722446 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722450 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722453 2581 flags.go:64] FLAG: --seccomp-default="false" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722456 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 21:12:54.046170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722459 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722462 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722465 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722468 2581 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722472 2581 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722475 2581 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722477 2581 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722480 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722483 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722486 2581 flags.go:64] FLAG: --system-cgroups="" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722489 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722494 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722496 2581 flags.go:64] FLAG: --tls-cert-file="" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722499 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722504 2581 flags.go:64] FLAG: --tls-min-version="" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722507 2581 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722509 2581 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722512 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722515 2581 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722519 2581 flags.go:64] FLAG: --v="2" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722523 2581 flags.go:64] FLAG: --version="false" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722527 2581 flags.go:64] FLAG: --vmodule="" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722530 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722534 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722670 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 21:12:54.046885 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722674 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 21:12:54.163784 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722677 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 21:12:54.163784 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722680 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 21:12:54.163784 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722684 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 21:12:54.163784 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722690 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 21:12:54.163784 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722694 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 21:12:54.163784 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722697 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 21:12:54.163784 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722700 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 21:12:54.163784 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722703 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 21:12:54.163784 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722706 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 21:12:54.163784 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722709 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 21:12:54.163784 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722712 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 21:12:54.163784 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722715 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 21:12:54.163784 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722717 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 21:12:54.163784 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722721 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 21:12:54.163784 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722723 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 21:12:54.163784 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722726 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 21:12:54.163784 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722729 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 21:12:54.163784 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722731 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 21:12:54.163784 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722734 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 21:12:54.164605 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722736 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 21:12:54.164605 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722739 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 21:12:54.164605 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722741 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 21:12:54.164605 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722744 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 21:12:54.164605 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722746 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 21:12:54.164605 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722749 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 21:12:54.164605 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722751 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 21:12:54.164605 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722755 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 21:12:54.164605 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722757 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 21:12:54.164605 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722761 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 21:12:54.164605 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722764 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 21:12:54.164605 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722767 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 21:12:54.164605 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722770 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 21:12:54.164605 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722773 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 21:12:54.164605 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722775 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 21:12:54.164605 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722779 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 21:12:54.164605 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722783 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 21:12:54.164605 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722786 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 21:12:54.164605 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722789 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 21:12:54.164605 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722792 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 21:12:54.165246 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722795 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 21:12:54.165246 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722798 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 21:12:54.165246 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722800 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 21:12:54.165246 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722803 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 21:12:54.165246 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722806 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 21:12:54.165246 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722809 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 21:12:54.165246 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722812 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 21:12:54.165246 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722815 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 21:12:54.165246 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722818 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 21:12:54.165246 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722821 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 21:12:54.165246 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722824 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 21:12:54.165246 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722826 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 21:12:54.165246 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722829 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 21:12:54.165246 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722832 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 21:12:54.165246 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722834 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 21:12:54.165246 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722838 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 21:12:54.165246 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722841 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 21:12:54.165246 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722843 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 21:12:54.165246 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722846 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 21:12:54.165246 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722848 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 21:12:54.165883 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722851 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 21:12:54.165883 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722854 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 21:12:54.165883 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722857 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 21:12:54.165883 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722859 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 21:12:54.165883 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722862 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 21:12:54.165883 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722864 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 21:12:54.165883 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722866 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 21:12:54.165883 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722870 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 21:12:54.165883 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722874 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 06 21:12:54.165883 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722877 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 21:12:54.165883 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722879 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 21:12:54.165883 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722882 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 21:12:54.165883 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722884 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 21:12:54.165883 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722886 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 21:12:54.165883 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722889 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 21:12:54.165883 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722891 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 21:12:54.165883 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722894 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 21:12:54.165883 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722896 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 21:12:54.165883 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722899 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 21:12:54.165883 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722902 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 21:12:54.166520 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722904 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 21:12:54.166520 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722906 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 21:12:54.166520 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722909 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 21:12:54.166520 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722911 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 21:12:54.166520 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722914 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 21:12:54.166520 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722916 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 21:12:54.166520 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722919 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 21:12:54.166520 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722922 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 21:12:54.166520 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722924 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 21:12:54.166520 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722927 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 21:12:54.166520 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722931 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 21:12:54.166520 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722934 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 21:12:54.166520 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722937 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 21:12:54.166520 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722940 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 21:12:54.166520 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722943 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 21:12:54.166520 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722945 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 21:12:54.166520 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722948 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 21:12:54.166520 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.722950 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 21:12:54.167154 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.722955 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 21:12:54.167154 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.729803 2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 21:12:54.167154 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.729825 2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 21:12:54.167154 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.730915 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 21:12:54.167154 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.730922 2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 21:12:54.167154 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.730981 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 21:12:54.167154 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.730986 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 21:12:54.167154 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.730990 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 21:12:54.167154 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.730995 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 21:12:54.167154 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.730998 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 21:12:54.167154 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731001 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 21:12:54.167154 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731005 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 21:12:54.167154 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731008 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 21:12:54.167154 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731011 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 21:12:54.167580 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731013 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 21:12:54.167580 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731016 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 21:12:54.167580 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731018 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 21:12:54.167580 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731021 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 21:12:54.167580 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731023 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 21:12:54.167580 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731027 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 21:12:54.167580 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731031 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 21:12:54.167580 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731034 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 21:12:54.167580 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731037 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 21:12:54.167580 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731040 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 21:12:54.167580 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731042 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 21:12:54.167580 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731045 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 21:12:54.167580 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731048 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 21:12:54.167580 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731051 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 21:12:54.167580 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731054 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 21:12:54.167580 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731056 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 21:12:54.167580 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731059 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 21:12:54.167580 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731061 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 21:12:54.167580 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731063 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 21:12:54.167580 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731066 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 21:12:54.168218 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731071 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 21:12:54.168218 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731074 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 21:12:54.168218 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731076 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 21:12:54.168218 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731079 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 21:12:54.168218 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731081 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 21:12:54.168218 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731084 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 21:12:54.168218 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731086 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 21:12:54.168218 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731089 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 21:12:54.168218 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731091 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 21:12:54.168218 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731094 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 21:12:54.168218 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731096 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 21:12:54.168218 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731099 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 21:12:54.168218 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731101 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 21:12:54.168218 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731104 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 21:12:54.168218 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731107 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 21:12:54.168218 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731109 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 21:12:54.168218 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731112 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 21:12:54.168218 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731114 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 21:12:54.168218 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731117 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 21:12:54.168218 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731119 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 21:12:54.168892 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731122 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 21:12:54.168892 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731124 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 21:12:54.168892 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731127 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 21:12:54.168892 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731129 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 21:12:54.168892 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731132 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 21:12:54.168892 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731135 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 21:12:54.168892 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731138 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 21:12:54.168892 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731141 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 21:12:54.168892 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731143 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 21:12:54.168892 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731146 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 21:12:54.168892 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731148 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 21:12:54.168892 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731151 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 21:12:54.168892 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731155 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 21:12:54.168892 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731158 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 21:12:54.168892 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731160 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 21:12:54.168892 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731163 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 21:12:54.168892 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731165 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 21:12:54.168892 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731168 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 21:12:54.168892 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731171 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 21:12:54.168892 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731173 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 21:12:54.169594 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731175 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 21:12:54.169594 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731186 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 21:12:54.169594 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731189 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 21:12:54.169594 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731192 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 21:12:54.169594 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731194 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 21:12:54.169594 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731197 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 21:12:54.169594 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731199 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 21:12:54.169594 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731202 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 21:12:54.169594 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731205 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 21:12:54.169594 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731207 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 21:12:54.169594 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731210 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 21:12:54.169594 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731212 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 21:12:54.169594 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731215 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 21:12:54.169594 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731218 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 21:12:54.169594 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731220 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 21:12:54.169594 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731223 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 21:12:54.169594 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731226 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 21:12:54.169594 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731229 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 21:12:54.169594 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731231 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 21:12:54.169594 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731234 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 21:12:54.170235 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731237 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 21:12:54.170235 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731240 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 21:12:54.170235 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731242 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 21:12:54.170235 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731245 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 21:12:54.170235 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731250 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 21:12:54.170235 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731254 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 21:12:54.170235 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731257 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 21:12:54.170235 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731260 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 06 21:12:54.170235 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731263 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 21:12:54.170235 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731266 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 21:12:54.170235 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.731272 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 21:12:54.170235 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731400 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 21:12:54.170235 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731404 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 21:12:54.170235 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731407 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 21:12:54.170873 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731410 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 21:12:54.170873 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731413 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 21:12:54.170873 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731415 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 21:12:54.170873 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731418 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 21:12:54.170873 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731420 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 21:12:54.170873 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731423 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 21:12:54.170873 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731426 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 21:12:54.170873 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731428 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 21:12:54.170873 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731431 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 21:12:54.170873 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731434 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 21:12:54.170873 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731436 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 21:12:54.170873 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731439 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 21:12:54.170873 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731441 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 21:12:54.170873 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731444 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 21:12:54.170873 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731447 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 21:12:54.170873 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731450 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 21:12:54.170873 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731452 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 21:12:54.170873 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731455 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 21:12:54.170873 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731458 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 21:12:54.171735 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731460 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 21:12:54.171735 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731463 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 21:12:54.171735 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731466 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 21:12:54.171735 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731470 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 21:12:54.171735 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731474 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 21:12:54.171735 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731476 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 21:12:54.171735 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731479 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 21:12:54.171735 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731481 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 21:12:54.171735 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731484 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 21:12:54.171735 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731487 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 21:12:54.171735 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731489 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 21:12:54.171735 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731492 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 21:12:54.171735 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731494 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 21:12:54.171735 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731497 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 21:12:54.171735 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731499 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 21:12:54.171735 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731501 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 21:12:54.171735 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731504 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 21:12:54.171735 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731507 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 21:12:54.171735 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731509 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 21:12:54.171735 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731512 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 21:12:54.172390 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731514 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 21:12:54.172390 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731517 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 21:12:54.172390 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731519 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 21:12:54.172390 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731522 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 21:12:54.172390 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731524 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 21:12:54.172390 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731527 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 21:12:54.172390 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731529 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 06 21:12:54.172390 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731532 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 21:12:54.172390 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731534 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 21:12:54.172390 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731536 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 21:12:54.172390 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731539 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 21:12:54.172390 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731541 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 21:12:54.172390 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731544 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 21:12:54.172390 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731548 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 21:12:54.172390 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731552 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 21:12:54.172390 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731557 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 21:12:54.172390 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731562 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 21:12:54.172390 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731565 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 21:12:54.172390 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731568 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 21:12:54.173013 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731571 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 21:12:54.173013 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731573 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 21:12:54.173013 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731576 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 21:12:54.173013 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731578 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 21:12:54.173013 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731581 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 21:12:54.173013 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731584 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 21:12:54.173013 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731588 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 21:12:54.173013 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731591 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 21:12:54.173013 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731593 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 21:12:54.173013 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731596 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 21:12:54.173013 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731598 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 21:12:54.173013 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731602 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 21:12:54.173013 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731605 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 21:12:54.173013 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731608 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 21:12:54.173013 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731611 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 21:12:54.173013 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731614 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 21:12:54.173013 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731617 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 21:12:54.173013 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731619 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 21:12:54.173013 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731622 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 21:12:54.173646 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731624 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 21:12:54.173646 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731626 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 21:12:54.173646 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731629 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 21:12:54.173646 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731631 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 21:12:54.173646 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731634 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 21:12:54.173646 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731636 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 21:12:54.173646 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731639 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 21:12:54.173646 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731642 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 21:12:54.173646 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731644 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 21:12:54.173646 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731648 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 21:12:54.173646 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731652 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 21:12:54.173646 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731654 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 21:12:54.173646 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731657 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 21:12:54.173646 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731660 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 21:12:54.173646 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731662 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 21:12:54.173646 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731665 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 21:12:54.173646 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731667 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 21:12:54.173646 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731669 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 21:12:54.173646 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:53.731672 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 21:12:54.174270 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.731677 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 21:12:54.174270 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.731795 2581 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 21:12:54.174270 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.734172 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 21:12:54.174270 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.736544 2581 server.go:1019] "Starting client certificate rotation" Jul 06 21:12:54.174270 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.736645 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 21:12:54.174270 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.737534 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 21:12:54.174270 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.739253 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 21:12:54.174270 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.741582 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 21:12:54.174270 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.755679 2581 log.go:25] "Validated CRI v1 runtime API" Jul 06 21:12:54.174270 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.757957 2581 log.go:25] "Validated CRI v1 image API" Jul 06 21:12:54.174270 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.758981 2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 21:12:54.174270 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.764182 2581 fs.go:135] Filesystem UUIDs: map[481538a0-f505-4c30-80d2-7f1f34bff76e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a2a40274-9026-4489-89f9-fc0dbbca0704:/dev/nvme0n1p4] Jul 06 21:12:54.174677 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.764197 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 21:12:54.174744 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.770428 2581 manager.go:216] Machine: {Timestamp:2026-07-06 21:12:53.769027927 +0000 UTC m=+0.353899023 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100616 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec282a06bfdee330dd69dfac51f17f46 SystemUUID:ec282a06-bfde-e330-dd69-dfac51f17f46 BootID:4820d91b-ab41-426d-8184-1f13f2ea6841 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e3:74:06:5c:d1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e3:74:06:5c:d1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:04:aa:ba:a9:d8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 21:12:54.174744 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.770531 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 21:12:54.174744 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.770612 2581 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 21:12:54.174744 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.771679 2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 21:12:54.174744 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.771712 2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-54.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 21:12:54.174744 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.773191 2581 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 21:12:54.174744 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.773204 2581 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 21:12:54.174744 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.773213 2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.774034 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.774637 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.775771 2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.779550 2581 state_mem.go:36] "Initialized new in-memory state store" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.779668 2581 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.779697 2581 kubelet.go:486] "Attempting to sync node with API server" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.779712 2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.779726 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.779735 2581 kubelet.go:398] "Adding apiserver pod source" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.779743 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.780175 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.780197 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.784131 2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.787045 2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.787072 2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.788337 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.788350 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.788356 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.788362 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.788375 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.788399 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.788412 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.788419 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.788439 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.788446 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.788463 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.788468 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.789082 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.789090 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.792302 2581 server.go:1268] "Started kubelet" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:53.792335 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-54.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:53.792335 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.792393 2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.792412 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.792568 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.797145 2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.798174 2581 server.go:310] "Adding debug handlers to kubelet server" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:53.799643 2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 21:12:54.175055 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.803937 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-54.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:53.804233 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-54.ec2.internal.18bfcfb9aa222b66 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-54.ec2.internal,UID:ip-10-0-140-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-54.ec2.internal,},FirstTimestamp:2026-07-06 21:12:53.792271206 +0000 UTC m=+0.377142302,LastTimestamp:2026-07-06 21:12:53.792271206 +0000 UTC m=+0.377142302,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-54.ec2.internal,}" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.805345 2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.805365 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:53.805699 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.806110 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.806171 2581 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.806185 2581 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.806359 2581 reconstruct.go:97] "Volume reconstruction finished" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.806367 2581 reconciler.go:29] "Reconciler: start to sync state" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.806919 2581 factory.go:55] Registering systemd factory Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.806932 2581 factory.go:223] Registration of the systemd container factory successfully Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.807194 2581 factory.go:153] Registering CRI-O factory Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.807203 2581 factory.go:223] Registration of the crio container factory successfully Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.807253 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.807277 2581 factory.go:103] Registering Raw factory Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.807290 2581 manager.go:1196] Started watching for new ooms in manager Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.807681 2581 manager.go:349] Starting recovery of all containers Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:53.808772 2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-54.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:53.808860 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.814809 2581 manager.go:354] Recovery completed Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.818667 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.821694 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientMemory" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.821735 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.821746 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientPID" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.822312 2581 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.822324 2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.822339 2581 state_mem.go:36] "Initialized new in-memory state store" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.822318 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.823090 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientMemory" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.823113 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.823125 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientPID" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.823571 2581 policy_none.go:49] "None policy: Start" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.823587 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 21:12:54.178395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.823598 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.825104 2581 policy_none.go:47] "Start" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:53.826564 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-54.ec2.internal.18bfcfb9abe35e6d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-54.ec2.internal,UID:ip-10-0-140-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-54.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-54.ec2.internal,},FirstTimestamp:2026-07-06 21:12:53.821709933 +0000 UTC m=+0.406581029,LastTimestamp:2026-07-06 21:12:53.821709933 +0000 UTC m=+0.406581029,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-54.ec2.internal,}" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.834271 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tqs5d" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:53.838181 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-54.ec2.internal.18bfcfb9abe3d7bd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-54.ec2.internal,UID:ip-10-0-140-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-54.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-54.ec2.internal,},FirstTimestamp:2026-07-06 21:12:53.821740989 +0000 UTC m=+0.406612085,LastTimestamp:2026-07-06 21:12:53.821740989 +0000 UTC m=+0.406612085,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-54.ec2.internal,}" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.841753 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tqs5d" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:53.906723 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.908095 2581 manager.go:337] "Starting Device Plugin manager" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:53.908195 2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.908207 2581 server.go:87] "Starting device plugin registration server" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.908522 2581 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.908535 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.908635 2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.908723 2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.908733 2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:53.909239 2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:53.909290 2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-54.ec2.internal\" not found" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.957180 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.958400 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.958435 2581 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.958457 2581 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:53.958491 2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:53.960923 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.009375 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.010392 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientMemory" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.010420 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 21:12:54.179623 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.010431 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientPID" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.010456 2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-54.ec2.internal" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:54.017582 2581 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-54.ec2.internal\" not found" node="ip-10-0-140-54.ec2.internal" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.019512 2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-54.ec2.internal" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:54.019531 2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-54.ec2.internal\": node \"ip-10-0-140-54.ec2.internal\" not found" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:54.044145 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.059157 2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-54.ec2.internal"] Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.059270 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.060783 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientMemory" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.060804 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.060815 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientPID" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.063548 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.063735 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.063768 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.064402 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientMemory" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.064426 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.064436 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientPID" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.064660 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientMemory" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.064674 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.064686 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientPID" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.066068 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-54.ec2.internal" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.066094 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.066815 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientMemory" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.066835 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.066847 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientPID" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.108213 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b78cd9a8597733097f5408183b6fd42d-config\") pod \"kube-apiserver-proxy-ip-10-0-140-54.ec2.internal\" (UID: \"b78cd9a8597733097f5408183b6fd42d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-54.ec2.internal" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:54.108224 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-54.ec2.internal\" not found" node="ip-10-0-140-54.ec2.internal" Jul 06 21:12:54.317424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.108250 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4316e771519ebb81207605a3b9372f6b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal\" (UID: \"4316e771519ebb81207605a3b9372f6b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal" Jul 06 21:12:54.318262 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.108290 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4316e771519ebb81207605a3b9372f6b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal\" (UID: \"4316e771519ebb81207605a3b9372f6b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal" Jul 06 21:12:54.318262 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:54.112076 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-54.ec2.internal\" not found" node="ip-10-0-140-54.ec2.internal" Jul 06 21:12:54.318262 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:54.144812 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found" Jul 06 21:12:54.318262 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.208834 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b78cd9a8597733097f5408183b6fd42d-config\") pod \"kube-apiserver-proxy-ip-10-0-140-54.ec2.internal\" (UID: \"b78cd9a8597733097f5408183b6fd42d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-54.ec2.internal" Jul 06 21:12:54.318262 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.208867 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4316e771519ebb81207605a3b9372f6b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal\" (UID: \"4316e771519ebb81207605a3b9372f6b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal" Jul 06 21:12:54.318262 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.208893 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4316e771519ebb81207605a3b9372f6b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal\" (UID: \"4316e771519ebb81207605a3b9372f6b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal" Jul 06 21:12:54.318262 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.208940 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4316e771519ebb81207605a3b9372f6b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal\" (UID: \"4316e771519ebb81207605a3b9372f6b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal" Jul 06 21:12:54.318262 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.208947 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b78cd9a8597733097f5408183b6fd42d-config\") pod \"kube-apiserver-proxy-ip-10-0-140-54.ec2.internal\" (UID: \"b78cd9a8597733097f5408183b6fd42d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-54.ec2.internal" Jul 06 21:12:54.318262 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.208947 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4316e771519ebb81207605a3b9372f6b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal\" (UID: \"4316e771519ebb81207605a3b9372f6b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal" Jul 06 21:12:54.318262 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:54.245105 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found" Jul 06 21:12:54.429749 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:54.345768 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found" Jul 06 21:12:54.429749 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.411239 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal" Jul 06 21:12:54.429749 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:54.416267 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-54.ec2.internal" Jul 06 21:12:54.551574 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:54.446334 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found" Jul 06 21:12:54.551574 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:54.546843 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found" Jul 06 21:12:56.526572 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:56.469444 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found" Jul 06 21:12:56.526572 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:56.525247 2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 21:12:56.651309 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:56.558823 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 21:12:56.651309 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:56.558985 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 21:12:56.651309 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:56.559006 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 21:12:56.651309 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:56.570105 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found" Jul 06 21:12:56.756008 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:56.666194 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 21:07:55 +0000 UTC" deadline="2028-04-21 00:44:16.337222218 +0000 UTC" Jul 06 21:12:56.756008 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:56.666218 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15699h31m19.671007267s" Jul 06 21:12:56.756008 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:56.670335 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found" Jul 06 21:12:56.756008 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:56.679411 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 21:12:56.756008 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:56.687112 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientMemory" Jul 06 21:12:56.756008 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:56.687157 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 21:12:56.756008 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:56.687174 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientPID" Jul 06 21:12:56.890819 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:56.770615 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found" Jul 06 21:12:56.890819 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:56.808497 2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 21:12:56.890819 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:56.810242 2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 21:12:56.890819 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:56.821361 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4316e771519ebb81207605a3b9372f6b.slice/crio-81aa7c2d95ee2550f5696ea7d81788db8601d2b43f02c56bddc35bd034a34322 WatchSource:0}: Error finding container 81aa7c2d95ee2550f5696ea7d81788db8601d2b43f02c56bddc35bd034a34322: Status 404 returned error can't find the container with id 81aa7c2d95ee2550f5696ea7d81788db8601d2b43f02c56bddc35bd034a34322 Jul 06 21:12:56.890819 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:56.821951 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78cd9a8597733097f5408183b6fd42d.slice/crio-d55089fa0552646987373ac5f77d43295007db520a019d50e6e5b485d6cdf497 WatchSource:0}: Error finding container d55089fa0552646987373ac5f77d43295007db520a019d50e6e5b485d6cdf497: Status 404 returned error can't find the container with id d55089fa0552646987373ac5f77d43295007db520a019d50e6e5b485d6cdf497 Jul 06 21:12:56.890819 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:56.827748 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 21:12:56.890819 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:56.827880 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal" Jul 06 21:12:56.890819 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:56.845170 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 21:12:56.890819 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:56.846134 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-54.ec2.internal" Jul 06 21:12:56.890819 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:56.858052 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 21:12:57.611372 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.538682 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 21:12:57.611372 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.602699 2581 apiserver.go:52] "Watching apiserver" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.616678 2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.618781 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-lds24","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal","openshift-multus/multus-additional-cni-plugins-x4grf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl","openshift-multus/multus-xq8wc","openshift-multus/network-metrics-daemon-79rs9","openshift-network-diagnostics/network-check-target-wjc92","openshift-network-operator/iptables-alerter-nqzqb","openshift-ovn-kubernetes/ovnkube-node-fq68j","kube-system/global-pull-secret-syncer-zn2lg","kube-system/konnectivity-agent-mxfzl","kube-system/kube-apiserver-proxy-ip-10-0-140-54.ec2.internal","openshift-cluster-node-tuning-operator/tuned-xngg9","openshift-dns/node-resolver-2q95f"] Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.620437 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:57.620503 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79rs9" podUID="e4f4e605-6394-4afd-8ef5-55c6fe78cb07" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.621861 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.623199 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.624328 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.624488 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qq67k\"" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.624621 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.624792 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mxfzl" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.626415 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.627573 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:57.627659 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjc92" podUID="b4d5a450-11db-44a5-bb22-35f668de29c7" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.628694 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.628747 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.629029 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.629235 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lds24" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.629323 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.629690 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.629768 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.630261 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.630265 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d4qxs\"" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.630269 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zftbz\"" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.630553 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 21:12:57.672754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.630647 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ln4tj\"" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.630857 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nqzqb" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.631078 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.632243 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.633010 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.633332 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.633502 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.633831 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j2htp\"" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.633833 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.634021 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-thztn\"" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:57.634043 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zn2lg" podUID="e4aacd25-63b3-4414-8e53-5d418169367b" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.634959 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.635055 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.635077 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.635387 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.635662 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.635713 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7rm8s\"" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.636699 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2q95f" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.638026 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.638064 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.638079 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.638539 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-etc-systemd\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.638560 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-lib-modules\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.638586 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d38a1184-76ff-4dc4-bacf-bbf90f547ca4-agent-certs\") pod \"konnectivity-agent-mxfzl\" (UID: \"d38a1184-76ff-4dc4-bacf-bbf90f547ca4\") " pod="kube-system/konnectivity-agent-mxfzl" Jul 06 21:12:57.674284 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.638607 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-host-run-k8s-cni-cncf-io\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.677800 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.638641 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-host-var-lib-cni-multus\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.677800 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.638664 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-hostroot\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.677800 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.638696 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0eafbe5d-cd16-448b-b3c0-156ca0c03981-multus-daemon-config\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.677800 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.638731 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08f54334-35ee-460d-ae74-75618e3e13c5-host\") pod \"node-ca-lds24\" (UID: \"08f54334-35ee-460d-ae74-75618e3e13c5\") " pod="openshift-image-registry/node-ca-lds24" Jul 06 21:12:57.677800 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.638757 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-etc-modprobe-d\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.677800 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.638790 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-etc-kubernetes\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.677800 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.638812 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-etc-sysctl-d\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.677800 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.638832 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-sys\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.677800 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.638860 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0eafbe5d-cd16-448b-b3c0-156ca0c03981-cni-binary-copy\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.677800 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.638884 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p798q\" (UniqueName: \"kubernetes.io/projected/08f54334-35ee-460d-ae74-75618e3e13c5-kube-api-access-p798q\") pod \"node-ca-lds24\" (UID: \"08f54334-35ee-460d-ae74-75618e3e13c5\") " pod="openshift-image-registry/node-ca-lds24" Jul 06 21:12:57.677800 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.638907 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs\") pod \"network-metrics-daemon-79rs9\" (UID: \"e4f4e605-6394-4afd-8ef5-55c6fe78cb07\") " pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:12:57.677800 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.638941 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.677800 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.638964 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-etc-sysctl-conf\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.677800 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.638988 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg78z\" (UniqueName: \"kubernetes.io/projected/0eafbe5d-cd16-448b-b3c0-156ca0c03981-kube-api-access-zg78z\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.678633 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639022 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-system-cni-dir\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.678633 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639049 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-cnibin\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.678633 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639072 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvbql\" (UniqueName: \"kubernetes.io/projected/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-kube-api-access-qvbql\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.678633 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639119 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z8nh\" (UniqueName: \"kubernetes.io/projected/e27e6563-2401-444f-962e-575b5e9cb93f-kube-api-access-7z8nh\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.678633 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639141 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-os-release\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.678633 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639162 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-host-var-lib-kubelet\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.678633 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639185 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-host-run-multus-certs\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.678633 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639209 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/08f54334-35ee-460d-ae74-75618e3e13c5-serviceca\") pod \"node-ca-lds24\" (UID: \"08f54334-35ee-460d-ae74-75618e3e13c5\") " pod="openshift-image-registry/node-ca-lds24" Jul 06 21:12:57.678633 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639232 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-cni-binary-copy\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.678633 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639259 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.678633 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639286 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-host\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.678633 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639302 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-system-cni-dir\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.678633 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639400 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-cnibin\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.678633 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639446 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-os-release\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.680000 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639471 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 21:12:57.680000 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639476 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw798\" (UniqueName: \"kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798\") pod \"network-check-target-wjc92\" (UID: \"b4d5a450-11db-44a5-bb22-35f668de29c7\") " pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:12:57.680000 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639501 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d38a1184-76ff-4dc4-bacf-bbf90f547ca4-konnectivity-ca\") pod \"konnectivity-agent-mxfzl\" (UID: \"d38a1184-76ff-4dc4-bacf-bbf90f547ca4\") " pod="kube-system/konnectivity-agent-mxfzl" Jul 06 21:12:57.680000 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639554 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-host-run-netns\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.680000 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639586 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-host-var-lib-cni-bin\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.680000 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639611 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c44sl\" (UniqueName: \"kubernetes.io/projected/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-kube-api-access-c44sl\") pod \"network-metrics-daemon-79rs9\" (UID: \"e4f4e605-6394-4afd-8ef5-55c6fe78cb07\") " pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:12:57.680000 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639640 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-run\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.680000 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639661 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 21:12:57.680000 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639680 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-var-lib-kubelet\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.680000 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639703 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e27e6563-2401-444f-962e-575b5e9cb93f-tmp\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.680000 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639719 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9hx8t\"" Jul 06 21:12:57.680000 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639726 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-multus-socket-dir-parent\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.680000 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639749 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-multus-conf-dir\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.680000 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639770 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-etc-kubernetes\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.680000 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639803 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.680979 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639828 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-etc-sysconfig\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.680979 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639836 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 21:12:57.680979 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639847 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e27e6563-2401-444f-962e-575b5e9cb93f-etc-tuned\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.680979 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639868 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-multus-cni-dir\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.680979 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.639905 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 21:12:57.680979 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.640116 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 21:12:57.680979 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.640388 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 21:12:57.680979 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.640717 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wrczr\"" Jul 06 21:12:57.680979 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.640915 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 21:12:57.680979 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.643688 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 21:12:57.680979 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.674210 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vcbzp" Jul 06 21:12:57.684251 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.682594 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vcbzp" Jul 06 21:12:57.728858 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.728835 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 21:12:57.740608 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.740577 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z8nh\" (UniqueName: \"kubernetes.io/projected/e27e6563-2401-444f-962e-575b5e9cb93f-kube-api-access-7z8nh\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.740608 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.740618 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-os-release\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.740803 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.740641 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-host-var-lib-kubelet\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.740803 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.740663 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-cni-binary-copy\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.740803 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.740688 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw798\" (UniqueName: \"kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798\") pod \"network-check-target-wjc92\" (UID: \"b4d5a450-11db-44a5-bb22-35f668de29c7\") " pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:12:57.740803 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.740714 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e4aacd25-63b3-4414-8e53-5d418169367b-dbus\") pod \"global-pull-secret-syncer-zn2lg\" (UID: \"e4aacd25-63b3-4414-8e53-5d418169367b\") " pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:12:57.740803 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.740728 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-os-release\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.740803 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.740736 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-node-log\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.740803 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.740760 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-host-var-lib-kubelet\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.740803 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.740792 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-host\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.741209 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.740821 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-system-cni-dir\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.741209 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.740854 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.741209 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.740881 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-ovnkube-config\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.741209 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.740905 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dvzs\" (UniqueName: \"kubernetes.io/projected/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-kube-api-access-4dvzs\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.741209 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.740930 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-host-run-netns\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.741209 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.740953 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-host-var-lib-cni-bin\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.741209 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.740981 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-host\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.741209 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.740984 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-host-slash\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.741209 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741047 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-system-cni-dir\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.741209 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741057 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-host-run-netns\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.741209 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741063 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-host-run-netns\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.741209 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741104 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-run-openvswitch\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.741209 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741112 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-host-var-lib-cni-bin\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.741209 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741129 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/50de3178-31c9-46a1-9f74-da997e92302a-sys-fs\") pod \"aws-ebs-csi-driver-node-qvtcl\" (UID: \"50de3178-31c9-46a1-9f74-da997e92302a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:57.741897 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741181 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-var-lib-kubelet\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.741897 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741221 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.741897 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741278 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-var-lib-kubelet\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.741897 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741280 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-systemd-units\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.741897 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741294 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-cni-binary-copy\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.741897 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741385 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-host-cni-netd\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.741897 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741401 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.741897 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741413 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbdnf\" (UniqueName: \"kubernetes.io/projected/50de3178-31c9-46a1-9f74-da997e92302a-kube-api-access-gbdnf\") pod \"aws-ebs-csi-driver-node-qvtcl\" (UID: \"50de3178-31c9-46a1-9f74-da997e92302a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:57.741897 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741455 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febe62f7-bf85-42c1-8902-c4dabb9ba553-tmp-dir\") pod \"node-resolver-2q95f\" (UID: \"febe62f7-bf85-42c1-8902-c4dabb9ba553\") " pod="openshift-dns/node-resolver-2q95f" Jul 06 21:12:57.741897 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741504 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret\") pod \"global-pull-secret-syncer-zn2lg\" (UID: \"e4aacd25-63b3-4414-8e53-5d418169367b\") " pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:12:57.741897 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741541 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-ovn-node-metrics-cert\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.741897 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741568 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-lib-modules\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.741897 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741592 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d38a1184-76ff-4dc4-bacf-bbf90f547ca4-agent-certs\") pod \"konnectivity-agent-mxfzl\" (UID: \"d38a1184-76ff-4dc4-bacf-bbf90f547ca4\") " pod="kube-system/konnectivity-agent-mxfzl" Jul 06 21:12:57.741897 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741641 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-host-var-lib-cni-multus\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.742354 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741664 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-hostroot\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.742354 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741687 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p798q\" (UniqueName: \"kubernetes.io/projected/08f54334-35ee-460d-ae74-75618e3e13c5-kube-api-access-p798q\") pod \"node-ca-lds24\" (UID: \"08f54334-35ee-460d-ae74-75618e3e13c5\") " pod="openshift-image-registry/node-ca-lds24" Jul 06 21:12:57.742354 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741710 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs\") pod \"network-metrics-daemon-79rs9\" (UID: \"e4f4e605-6394-4afd-8ef5-55c6fe78cb07\") " pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:12:57.742354 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741738 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.742354 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741763 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-log-socket\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.742354 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741764 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-lib-modules\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.742354 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741786 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-etc-sysctl-d\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.742354 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741810 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-sys\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.742354 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741835 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0eafbe5d-cd16-448b-b3c0-156ca0c03981-cni-binary-copy\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.742354 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741857 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-host-kubelet\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.742354 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741880 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/50de3178-31c9-46a1-9f74-da997e92302a-socket-dir\") pod \"aws-ebs-csi-driver-node-qvtcl\" (UID: \"50de3178-31c9-46a1-9f74-da997e92302a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:57.742354 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741904 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/50de3178-31c9-46a1-9f74-da997e92302a-registration-dir\") pod \"aws-ebs-csi-driver-node-qvtcl\" (UID: \"50de3178-31c9-46a1-9f74-da997e92302a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:57.742354 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741944 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/50de3178-31c9-46a1-9f74-da997e92302a-device-dir\") pod \"aws-ebs-csi-driver-node-qvtcl\" (UID: \"50de3178-31c9-46a1-9f74-da997e92302a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:57.742354 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741969 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/febe62f7-bf85-42c1-8902-c4dabb9ba553-hosts-file\") pod \"node-resolver-2q95f\" (UID: \"febe62f7-bf85-42c1-8902-c4dabb9ba553\") " pod="openshift-dns/node-resolver-2q95f" Jul 06 21:12:57.742978 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.741993 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-system-cni-dir\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.742978 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.742017 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-cnibin\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.742978 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.742071 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thlwm\" (UniqueName: \"kubernetes.io/projected/75b9c669-e2a1-4c31-b58c-905b30293f0a-kube-api-access-thlwm\") pod \"iptables-alerter-nqzqb\" (UID: \"75b9c669-e2a1-4c31-b58c-905b30293f0a\") " pod="openshift-network-operator/iptables-alerter-nqzqb" Jul 06 21:12:57.742978 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.742097 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-host-run-multus-certs\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.742978 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.742122 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/08f54334-35ee-460d-ae74-75618e3e13c5-serviceca\") pod \"node-ca-lds24\" (UID: \"08f54334-35ee-460d-ae74-75618e3e13c5\") " pod="openshift-image-registry/node-ca-lds24" Jul 06 21:12:57.742978 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.742145 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-host-cni-bin\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.742978 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.742174 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k687v\" (UniqueName: \"kubernetes.io/projected/febe62f7-bf85-42c1-8902-c4dabb9ba553-kube-api-access-k687v\") pod \"node-resolver-2q95f\" (UID: \"febe62f7-bf85-42c1-8902-c4dabb9ba553\") " pod="openshift-dns/node-resolver-2q95f" Jul 06 21:12:57.742978 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.742197 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-cnibin\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.742978 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.742220 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-os-release\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.742978 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.742242 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-run-systemd\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.742978 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.742267 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/50de3178-31c9-46a1-9f74-da997e92302a-etc-selinux\") pod \"aws-ebs-csi-driver-node-qvtcl\" (UID: \"50de3178-31c9-46a1-9f74-da997e92302a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:57.742978 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.742291 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d38a1184-76ff-4dc4-bacf-bbf90f547ca4-konnectivity-ca\") pod \"konnectivity-agent-mxfzl\" (UID: \"d38a1184-76ff-4dc4-bacf-bbf90f547ca4\") " pod="kube-system/konnectivity-agent-mxfzl" Jul 06 21:12:57.742978 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.742353 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c44sl\" (UniqueName: \"kubernetes.io/projected/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-kube-api-access-c44sl\") pod \"network-metrics-daemon-79rs9\" (UID: \"e4f4e605-6394-4afd-8ef5-55c6fe78cb07\") " pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:12:57.742978 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.742327 2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 21:12:57.743696 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.742381 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-env-overrides\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.743696 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:57.742398 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 21:12:57.743696 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:57.742475 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs podName:e4f4e605-6394-4afd-8ef5-55c6fe78cb07 nodeName:}" failed. No retries permitted until 2026-07-06 21:12:58.242454839 +0000 UTC m=+3.005367840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs") pod "network-metrics-daemon-79rs9" (UID: "e4f4e605-6394-4afd-8ef5-55c6fe78cb07") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 21:12:57.743696 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.742526 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-run\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.743696 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.742594 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-system-cni-dir\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.743696 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.742637 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-cnibin\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.743696 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.742689 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-host-run-multus-certs\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.743696 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.742953 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-host-var-lib-cni-multus\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.743696 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743045 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-cnibin\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.743696 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743138 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/08f54334-35ee-460d-ae74-75618e3e13c5-serviceca\") pod \"node-ca-lds24\" (UID: \"08f54334-35ee-460d-ae74-75618e3e13c5\") " pod="openshift-image-registry/node-ca-lds24" Jul 06 21:12:57.743696 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743172 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-os-release\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.743696 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743194 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.743696 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743217 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-hostroot\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.743696 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.742404 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-run\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.744323 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743279 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e27e6563-2401-444f-962e-575b5e9cb93f-tmp\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.744323 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743307 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-multus-socket-dir-parent\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.744323 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743324 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-etc-sysctl-d\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.744323 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743331 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-multus-conf-dir\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.744323 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743373 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-etc-kubernetes\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.744323 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743397 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-etc-sysconfig\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.744323 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743421 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e27e6563-2401-444f-962e-575b5e9cb93f-etc-tuned\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.744323 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743444 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-multus-cni-dir\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.744323 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743473 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/75b9c669-e2a1-4c31-b58c-905b30293f0a-iptables-alerter-script\") pod \"iptables-alerter-nqzqb\" (UID: \"75b9c669-e2a1-4c31-b58c-905b30293f0a\") " pod="openshift-network-operator/iptables-alerter-nqzqb" Jul 06 21:12:57.744323 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743498 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75b9c669-e2a1-4c31-b58c-905b30293f0a-host-slash\") pod \"iptables-alerter-nqzqb\" (UID: \"75b9c669-e2a1-4c31-b58c-905b30293f0a\") " pod="openshift-network-operator/iptables-alerter-nqzqb" Jul 06 21:12:57.744323 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743521 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-etc-openvswitch\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.744323 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743553 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/50de3178-31c9-46a1-9f74-da997e92302a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qvtcl\" (UID: \"50de3178-31c9-46a1-9f74-da997e92302a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:57.744323 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743398 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-sys\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.744323 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743576 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-etc-systemd\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.744323 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743621 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-etc-systemd\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.745070 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743612 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-host-run-k8s-cni-cncf-io\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.745070 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743648 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0eafbe5d-cd16-448b-b3c0-156ca0c03981-multus-daemon-config\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.745070 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743653 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d38a1184-76ff-4dc4-bacf-bbf90f547ca4-konnectivity-ca\") pod \"konnectivity-agent-mxfzl\" (UID: \"d38a1184-76ff-4dc4-bacf-bbf90f547ca4\") " pod="kube-system/konnectivity-agent-mxfzl" Jul 06 21:12:57.745070 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743698 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08f54334-35ee-460d-ae74-75618e3e13c5-host\") pod \"node-ca-lds24\" (UID: \"08f54334-35ee-460d-ae74-75618e3e13c5\") " pod="openshift-image-registry/node-ca-lds24" Jul 06 21:12:57.745070 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743704 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-host-run-k8s-cni-cncf-io\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.745070 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743708 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-multus-conf-dir\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.745070 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743718 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-host-run-ovn-kubernetes\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.745070 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743735 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-ovnkube-script-lib\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.745070 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743750 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-etc-modprobe-d\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.745070 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743765 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-etc-kubernetes\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.745070 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743781 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.745070 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743796 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-var-lib-openvswitch\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.745070 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743812 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-run-ovn\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.745070 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743827 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-etc-sysctl-conf\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.745757 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743845 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg78z\" (UniqueName: \"kubernetes.io/projected/0eafbe5d-cd16-448b-b3c0-156ca0c03981-kube-api-access-zg78z\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.745757 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743903 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvbql\" (UniqueName: \"kubernetes.io/projected/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-kube-api-access-qvbql\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.745757 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743907 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0eafbe5d-cd16-448b-b3c0-156ca0c03981-cni-binary-copy\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.745757 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.743919 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e4aacd25-63b3-4414-8e53-5d418169367b-kubelet-config\") pod \"global-pull-secret-syncer-zn2lg\" (UID: \"e4aacd25-63b3-4414-8e53-5d418169367b\") " pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:12:57.745757 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.744028 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-multus-cni-dir\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.745757 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.744154 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-etc-kubernetes\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.745757 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.744169 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0eafbe5d-cd16-448b-b3c0-156ca0c03981-multus-daemon-config\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.745757 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.744267 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-etc-modprobe-d\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.745757 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.744293 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-etc-kubernetes\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.745757 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.744307 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0eafbe5d-cd16-448b-b3c0-156ca0c03981-multus-socket-dir-parent\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.745757 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.744368 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-etc-sysconfig\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.745757 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.744390 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e27e6563-2401-444f-962e-575b5e9cb93f-etc-sysctl-conf\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.745757 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.744402 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08f54334-35ee-460d-ae74-75618e3e13c5-host\") pod \"node-ca-lds24\" (UID: \"08f54334-35ee-460d-ae74-75618e3e13c5\") " pod="openshift-image-registry/node-ca-lds24" Jul 06 21:12:57.745757 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.744748 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.746388 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.746269 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d38a1184-76ff-4dc4-bacf-bbf90f547ca4-agent-certs\") pod \"konnectivity-agent-mxfzl\" (UID: \"d38a1184-76ff-4dc4-bacf-bbf90f547ca4\") " pod="kube-system/konnectivity-agent-mxfzl" Jul 06 21:12:57.746633 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.746581 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e27e6563-2401-444f-962e-575b5e9cb93f-tmp\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.747897 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:57.747778 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 21:12:57.747897 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:57.747801 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 21:12:57.747897 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:57.747814 2581 projected.go:196] Error preparing data for projected volume kube-api-access-pw798 for pod openshift-network-diagnostics/network-check-target-wjc92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 21:12:57.747897 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:57.747875 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798 podName:b4d5a450-11db-44a5-bb22-35f668de29c7 nodeName:}" failed. No retries permitted until 2026-07-06 21:12:58.247858948 +0000 UTC m=+3.010771953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pw798" (UniqueName: "kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798") pod "network-check-target-wjc92" (UID: "b4d5a450-11db-44a5-bb22-35f668de29c7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 21:12:57.748333 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.748304 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e27e6563-2401-444f-962e-575b5e9cb93f-etc-tuned\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.751966 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.751766 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z8nh\" (UniqueName: \"kubernetes.io/projected/e27e6563-2401-444f-962e-575b5e9cb93f-kube-api-access-7z8nh\") pod \"tuned-xngg9\" (UID: \"e27e6563-2401-444f-962e-575b5e9cb93f\") " pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.751966 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.751923 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p798q\" (UniqueName: \"kubernetes.io/projected/08f54334-35ee-460d-ae74-75618e3e13c5-kube-api-access-p798q\") pod \"node-ca-lds24\" (UID: \"08f54334-35ee-460d-ae74-75618e3e13c5\") " pod="openshift-image-registry/node-ca-lds24" Jul 06 21:12:57.752945 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.752923 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvbql\" (UniqueName: \"kubernetes.io/projected/b03516a3-66d9-4cd4-b814-dc23dd3bd9a1-kube-api-access-qvbql\") pod \"multus-additional-cni-plugins-x4grf\" (UID: \"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1\") " pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.754096 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.754073 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c44sl\" (UniqueName: \"kubernetes.io/projected/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-kube-api-access-c44sl\") pod \"network-metrics-daemon-79rs9\" (UID: \"e4f4e605-6394-4afd-8ef5-55c6fe78cb07\") " pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:12:57.755695 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.755663 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg78z\" (UniqueName: \"kubernetes.io/projected/0eafbe5d-cd16-448b-b3c0-156ca0c03981-kube-api-access-zg78z\") pod \"multus-xq8wc\" (UID: \"0eafbe5d-cd16-448b-b3c0-156ca0c03981\") " pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.785950 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.785893 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-54.ec2.internal" event={"ID":"b78cd9a8597733097f5408183b6fd42d","Type":"ContainerStarted","Data":"d55089fa0552646987373ac5f77d43295007db520a019d50e6e5b485d6cdf497"} Jul 06 21:12:57.787504 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.787473 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal" event={"ID":"4316e771519ebb81207605a3b9372f6b","Type":"ContainerStarted","Data":"81aa7c2d95ee2550f5696ea7d81788db8601d2b43f02c56bddc35bd034a34322"} Jul 06 21:12:57.845789 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.845741 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thlwm\" (UniqueName: \"kubernetes.io/projected/75b9c669-e2a1-4c31-b58c-905b30293f0a-kube-api-access-thlwm\") pod \"iptables-alerter-nqzqb\" (UID: \"75b9c669-e2a1-4c31-b58c-905b30293f0a\") " pod="openshift-network-operator/iptables-alerter-nqzqb" Jul 06 21:12:57.845789 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.845795 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-host-cni-bin\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.846024 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.845818 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k687v\" (UniqueName: \"kubernetes.io/projected/febe62f7-bf85-42c1-8902-c4dabb9ba553-kube-api-access-k687v\") pod \"node-resolver-2q95f\" (UID: \"febe62f7-bf85-42c1-8902-c4dabb9ba553\") " pod="openshift-dns/node-resolver-2q95f" Jul 06 21:12:57.846024 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.845840 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-run-systemd\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.846024 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.845859 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/50de3178-31c9-46a1-9f74-da997e92302a-etc-selinux\") pod \"aws-ebs-csi-driver-node-qvtcl\" (UID: \"50de3178-31c9-46a1-9f74-da997e92302a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:57.846024 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.845881 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-env-overrides\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.846024 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.845910 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/75b9c669-e2a1-4c31-b58c-905b30293f0a-iptables-alerter-script\") pod \"iptables-alerter-nqzqb\" (UID: \"75b9c669-e2a1-4c31-b58c-905b30293f0a\") " pod="openshift-network-operator/iptables-alerter-nqzqb" Jul 06 21:12:57.846024 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.845932 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75b9c669-e2a1-4c31-b58c-905b30293f0a-host-slash\") pod \"iptables-alerter-nqzqb\" (UID: \"75b9c669-e2a1-4c31-b58c-905b30293f0a\") " pod="openshift-network-operator/iptables-alerter-nqzqb" Jul 06 21:12:57.846024 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.845952 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-etc-openvswitch\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.846024 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.845971 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/50de3178-31c9-46a1-9f74-da997e92302a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qvtcl\" (UID: \"50de3178-31c9-46a1-9f74-da997e92302a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:57.846024 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846000 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-host-run-ovn-kubernetes\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.846024 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846022 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-ovnkube-script-lib\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.846514 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846046 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-var-lib-openvswitch\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.846514 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846045 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-host-cni-bin\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.846514 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846071 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-run-ovn\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.846514 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846100 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e4aacd25-63b3-4414-8e53-5d418169367b-kubelet-config\") pod \"global-pull-secret-syncer-zn2lg\" (UID: \"e4aacd25-63b3-4414-8e53-5d418169367b\") " pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:12:57.846514 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846109 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75b9c669-e2a1-4c31-b58c-905b30293f0a-host-slash\") pod \"iptables-alerter-nqzqb\" (UID: \"75b9c669-e2a1-4c31-b58c-905b30293f0a\") " pod="openshift-network-operator/iptables-alerter-nqzqb" Jul 06 21:12:57.846514 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846169 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-host-run-ovn-kubernetes\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.846514 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846176 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e4aacd25-63b3-4414-8e53-5d418169367b-kubelet-config\") pod \"global-pull-secret-syncer-zn2lg\" (UID: \"e4aacd25-63b3-4414-8e53-5d418169367b\") " pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:12:57.846514 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846213 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-etc-openvswitch\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.846514 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846220 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/50de3178-31c9-46a1-9f74-da997e92302a-etc-selinux\") pod \"aws-ebs-csi-driver-node-qvtcl\" (UID: \"50de3178-31c9-46a1-9f74-da997e92302a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:57.846514 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846250 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/50de3178-31c9-46a1-9f74-da997e92302a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qvtcl\" (UID: \"50de3178-31c9-46a1-9f74-da997e92302a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:57.846514 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846285 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e4aacd25-63b3-4414-8e53-5d418169367b-dbus\") pod \"global-pull-secret-syncer-zn2lg\" (UID: \"e4aacd25-63b3-4414-8e53-5d418169367b\") " pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:12:57.846514 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846304 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-run-ovn\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.846514 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846318 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-node-log\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.846514 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846374 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.847191 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846400 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-ovnkube-config\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.847191 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846407 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-var-lib-openvswitch\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.847191 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846423 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dvzs\" (UniqueName: \"kubernetes.io/projected/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-kube-api-access-4dvzs\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.847191 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846441 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-node-log\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.847191 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846453 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.847191 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846448 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e4aacd25-63b3-4414-8e53-5d418169367b-dbus\") pod \"global-pull-secret-syncer-zn2lg\" (UID: \"e4aacd25-63b3-4414-8e53-5d418169367b\") " pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:12:57.847191 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846538 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-run-systemd\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.847191 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846580 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-host-slash\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.847191 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846609 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-env-overrides\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.847191 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846621 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-host-run-netns\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.847191 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846641 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-host-slash\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.847191 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846676 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-run-openvswitch\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.847191 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846699 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-host-run-netns\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.847191 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846741 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/50de3178-31c9-46a1-9f74-da997e92302a-sys-fs\") pod \"aws-ebs-csi-driver-node-qvtcl\" (UID: \"50de3178-31c9-46a1-9f74-da997e92302a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:57.847191 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846795 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/50de3178-31c9-46a1-9f74-da997e92302a-sys-fs\") pod \"aws-ebs-csi-driver-node-qvtcl\" (UID: \"50de3178-31c9-46a1-9f74-da997e92302a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:57.847891 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846806 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-ovnkube-script-lib\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.847891 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846824 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-run-openvswitch\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.847891 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846846 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/75b9c669-e2a1-4c31-b58c-905b30293f0a-iptables-alerter-script\") pod \"iptables-alerter-nqzqb\" (UID: \"75b9c669-e2a1-4c31-b58c-905b30293f0a\") " pod="openshift-network-operator/iptables-alerter-nqzqb" Jul 06 21:12:57.847891 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846856 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-systemd-units\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.847891 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846883 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-host-cni-netd\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.847891 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846883 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-ovnkube-config\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.847891 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846908 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbdnf\" (UniqueName: \"kubernetes.io/projected/50de3178-31c9-46a1-9f74-da997e92302a-kube-api-access-gbdnf\") pod \"aws-ebs-csi-driver-node-qvtcl\" (UID: \"50de3178-31c9-46a1-9f74-da997e92302a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:57.847891 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846935 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-systemd-units\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.847891 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846936 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febe62f7-bf85-42c1-8902-c4dabb9ba553-tmp-dir\") pod \"node-resolver-2q95f\" (UID: \"febe62f7-bf85-42c1-8902-c4dabb9ba553\") " pod="openshift-dns/node-resolver-2q95f" Jul 06 21:12:57.847891 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.846997 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret\") pod \"global-pull-secret-syncer-zn2lg\" (UID: \"e4aacd25-63b3-4414-8e53-5d418169367b\") " pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:12:57.847891 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.847116 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-ovn-node-metrics-cert\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.847891 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:57.847198 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 21:12:57.847891 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.847201 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febe62f7-bf85-42c1-8902-c4dabb9ba553-tmp-dir\") pod \"node-resolver-2q95f\" (UID: \"febe62f7-bf85-42c1-8902-c4dabb9ba553\") " pod="openshift-dns/node-resolver-2q95f" Jul 06 21:12:57.847891 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.847261 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-host-cni-netd\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.848569 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:57.847324 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret podName:e4aacd25-63b3-4414-8e53-5d418169367b nodeName:}" failed. No retries permitted until 2026-07-06 21:12:58.347292944 +0000 UTC m=+3.110205928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret") pod "global-pull-secret-syncer-zn2lg" (UID: "e4aacd25-63b3-4414-8e53-5d418169367b") : object "kube-system"/"original-pull-secret" not registered Jul 06 21:12:57.848569 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.847399 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-log-socket\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.848569 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.847428 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-host-kubelet\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.848569 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.847453 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/50de3178-31c9-46a1-9f74-da997e92302a-socket-dir\") pod \"aws-ebs-csi-driver-node-qvtcl\" (UID: \"50de3178-31c9-46a1-9f74-da997e92302a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:57.848569 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.847478 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/50de3178-31c9-46a1-9f74-da997e92302a-registration-dir\") pod \"aws-ebs-csi-driver-node-qvtcl\" (UID: \"50de3178-31c9-46a1-9f74-da997e92302a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:57.848569 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.847515 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/50de3178-31c9-46a1-9f74-da997e92302a-device-dir\") pod \"aws-ebs-csi-driver-node-qvtcl\" (UID: \"50de3178-31c9-46a1-9f74-da997e92302a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:57.848569 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.847540 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/febe62f7-bf85-42c1-8902-c4dabb9ba553-hosts-file\") pod \"node-resolver-2q95f\" (UID: \"febe62f7-bf85-42c1-8902-c4dabb9ba553\") " pod="openshift-dns/node-resolver-2q95f" Jul 06 21:12:57.848569 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.847554 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-host-kubelet\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.848569 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.847618 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/50de3178-31c9-46a1-9f74-da997e92302a-registration-dir\") pod \"aws-ebs-csi-driver-node-qvtcl\" (UID: \"50de3178-31c9-46a1-9f74-da997e92302a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:57.848569 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.847619 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/50de3178-31c9-46a1-9f74-da997e92302a-device-dir\") pod \"aws-ebs-csi-driver-node-qvtcl\" (UID: \"50de3178-31c9-46a1-9f74-da997e92302a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:57.848569 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.847669 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/50de3178-31c9-46a1-9f74-da997e92302a-socket-dir\") pod \"aws-ebs-csi-driver-node-qvtcl\" (UID: \"50de3178-31c9-46a1-9f74-da997e92302a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:57.848569 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.847710 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/febe62f7-bf85-42c1-8902-c4dabb9ba553-hosts-file\") pod \"node-resolver-2q95f\" (UID: \"febe62f7-bf85-42c1-8902-c4dabb9ba553\") " pod="openshift-dns/node-resolver-2q95f" Jul 06 21:12:57.848569 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.847748 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-log-socket\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.850005 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.849963 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-ovn-node-metrics-cert\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.856013 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.855596 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thlwm\" (UniqueName: \"kubernetes.io/projected/75b9c669-e2a1-4c31-b58c-905b30293f0a-kube-api-access-thlwm\") pod \"iptables-alerter-nqzqb\" (UID: \"75b9c669-e2a1-4c31-b58c-905b30293f0a\") " pod="openshift-network-operator/iptables-alerter-nqzqb" Jul 06 21:12:57.856013 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.855978 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k687v\" (UniqueName: \"kubernetes.io/projected/febe62f7-bf85-42c1-8902-c4dabb9ba553-kube-api-access-k687v\") pod \"node-resolver-2q95f\" (UID: \"febe62f7-bf85-42c1-8902-c4dabb9ba553\") " pod="openshift-dns/node-resolver-2q95f" Jul 06 21:12:57.856765 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.856736 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbdnf\" (UniqueName: \"kubernetes.io/projected/50de3178-31c9-46a1-9f74-da997e92302a-kube-api-access-gbdnf\") pod \"aws-ebs-csi-driver-node-qvtcl\" (UID: \"50de3178-31c9-46a1-9f74-da997e92302a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:57.857449 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.857421 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dvzs\" (UniqueName: \"kubernetes.io/projected/a276f91f-2f3c-4f2c-85ff-f53d4bc6c713-kube-api-access-4dvzs\") pod \"ovnkube-node-fq68j\" (UID: \"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:57.937458 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.937391 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xngg9" Jul 06 21:12:57.945025 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.944990 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x4grf" Jul 06 21:12:57.954452 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.954422 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mxfzl" Jul 06 21:12:57.964458 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.964433 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xq8wc" Jul 06 21:12:57.977459 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:57.977383 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lds24" Jul 06 21:12:58.013652 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:58.013620 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nqzqb" Jul 06 21:12:58.023076 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:58.023049 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:12:58.029173 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:58.029153 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" Jul 06 21:12:58.036037 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:58.036013 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2q95f" Jul 06 21:12:58.250649 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:58.250572 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw798\" (UniqueName: \"kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798\") pod \"network-check-target-wjc92\" (UID: \"b4d5a450-11db-44a5-bb22-35f668de29c7\") " pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:12:58.250649 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:58.250629 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs\") pod \"network-metrics-daemon-79rs9\" (UID: \"e4f4e605-6394-4afd-8ef5-55c6fe78cb07\") " pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:12:58.250857 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:58.250755 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 21:12:58.250857 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:58.250820 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs podName:e4f4e605-6394-4afd-8ef5-55c6fe78cb07 nodeName:}" failed. No retries permitted until 2026-07-06 21:12:59.250806054 +0000 UTC m=+4.013719036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs") pod "network-metrics-daemon-79rs9" (UID: "e4f4e605-6394-4afd-8ef5-55c6fe78cb07") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 21:12:58.250857 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:58.250757 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 21:12:58.250999 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:58.250875 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 21:12:58.250999 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:58.250889 2581 projected.go:196] Error preparing data for projected volume kube-api-access-pw798 for pod openshift-network-diagnostics/network-check-target-wjc92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 21:12:58.250999 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:58.250948 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798 podName:b4d5a450-11db-44a5-bb22-35f668de29c7 nodeName:}" failed. No retries permitted until 2026-07-06 21:12:59.250931369 +0000 UTC m=+4.013844368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pw798" (UniqueName: "kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798") pod "network-check-target-wjc92" (UID: "b4d5a450-11db-44a5-bb22-35f668de29c7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 21:12:58.351519 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:58.351484 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret\") pod \"global-pull-secret-syncer-zn2lg\" (UID: \"e4aacd25-63b3-4414-8e53-5d418169367b\") " pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:12:58.351685 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:58.351609 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 21:12:58.351685 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:58.351678 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret podName:e4aacd25-63b3-4414-8e53-5d418169367b nodeName:}" failed. No retries permitted until 2026-07-06 21:12:59.351659238 +0000 UTC m=+4.114572227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret") pod "global-pull-secret-syncer-zn2lg" (UID: "e4aacd25-63b3-4414-8e53-5d418169367b") : object "kube-system"/"original-pull-secret" not registered Jul 06 21:12:58.657915 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:58.657840 2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 21:12:58.684936 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:58.684875 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 21:07:57 +0000 UTC" deadline="2028-01-05 09:10:38.172384314 +0000 UTC" Jul 06 21:12:58.684936 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:58.684931 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13139h57m39.487457885s" Jul 06 21:12:58.885559 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:58.885520 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfebe62f7_bf85_42c1_8902_c4dabb9ba553.slice/crio-4670e09fe7e58829fa761a8d859829fb2b8b50e6cd8a2e22108e6aade1e3985d WatchSource:0}: Error finding container 4670e09fe7e58829fa761a8d859829fb2b8b50e6cd8a2e22108e6aade1e3985d: Status 404 returned error can't find the container with id 4670e09fe7e58829fa761a8d859829fb2b8b50e6cd8a2e22108e6aade1e3985d Jul 06 21:12:58.887097 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:58.887005 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f54334_35ee_460d_ae74_75618e3e13c5.slice/crio-d52fb3ab8ee93e4f3669d60fe271416181871c9d8daa933da67b0a42b7900afc WatchSource:0}: Error finding container d52fb3ab8ee93e4f3669d60fe271416181871c9d8daa933da67b0a42b7900afc: Status 404 returned error can't find the container with id d52fb3ab8ee93e4f3669d60fe271416181871c9d8daa933da67b0a42b7900afc Jul 06 21:12:58.891360 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:58.890328 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda276f91f_2f3c_4f2c_85ff_f53d4bc6c713.slice/crio-cf920acefb47d21bcfee8960c413c1abc93536e05d9a5e771167ebff0d374c88 WatchSource:0}: Error finding container cf920acefb47d21bcfee8960c413c1abc93536e05d9a5e771167ebff0d374c88: Status 404 returned error can't find the container with id cf920acefb47d21bcfee8960c413c1abc93536e05d9a5e771167ebff0d374c88 Jul 06 21:12:58.891563 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:58.891466 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50de3178_31c9_46a1_9f74_da997e92302a.slice/crio-f8993003bb829c766c88106dc09c39bc5bfbbb7774e35a341a7b4492339ea2af WatchSource:0}: Error finding container f8993003bb829c766c88106dc09c39bc5bfbbb7774e35a341a7b4492339ea2af: Status 404 returned error can't find the container with id f8993003bb829c766c88106dc09c39bc5bfbbb7774e35a341a7b4492339ea2af Jul 06 21:12:58.892648 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:58.891916 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eafbe5d_cd16_448b_b3c0_156ca0c03981.slice/crio-69c0cb590d4ecdf3f1491f15f1fc30afd9dbea38fb7a4d42a9390d141f380365 WatchSource:0}: Error finding container 69c0cb590d4ecdf3f1491f15f1fc30afd9dbea38fb7a4d42a9390d141f380365: Status 404 returned error can't find the container with id 69c0cb590d4ecdf3f1491f15f1fc30afd9dbea38fb7a4d42a9390d141f380365 Jul 06 21:12:58.892953 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:58.892913 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38a1184_76ff_4dc4_bacf_bbf90f547ca4.slice/crio-9ed08dd7010f6cd2e12deeb31e3974845d84b0363410f1234867732ea22f568e WatchSource:0}: Error finding container 9ed08dd7010f6cd2e12deeb31e3974845d84b0363410f1234867732ea22f568e: Status 404 returned error can't find the container with id 9ed08dd7010f6cd2e12deeb31e3974845d84b0363410f1234867732ea22f568e Jul 06 21:12:58.893986 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:58.893772 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b9c669_e2a1_4c31_b58c_905b30293f0a.slice/crio-5434b48c106d9af540067cd43cc681185577ba5895d89a6cc31780fbca4e18bc WatchSource:0}: Error finding container 5434b48c106d9af540067cd43cc681185577ba5895d89a6cc31780fbca4e18bc: Status 404 returned error can't find the container with id 5434b48c106d9af540067cd43cc681185577ba5895d89a6cc31780fbca4e18bc Jul 06 21:12:58.894662 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:58.894641 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27e6563_2401_444f_962e_575b5e9cb93f.slice/crio-4577e86eb1c123292f8d48ae4061ac3a75d2850a22eeb6a6747b9f5862902108 WatchSource:0}: Error finding container 4577e86eb1c123292f8d48ae4061ac3a75d2850a22eeb6a6747b9f5862902108: Status 404 returned error can't find the container with id 4577e86eb1c123292f8d48ae4061ac3a75d2850a22eeb6a6747b9f5862902108 Jul 06 21:12:58.896401 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:12:58.896381 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03516a3_66d9_4cd4_b814_dc23dd3bd9a1.slice/crio-3ac160b00f4333cba6787849aeeb9f7d09308e9bd252ed2bed3af7afead7a4fd WatchSource:0}: Error finding container 3ac160b00f4333cba6787849aeeb9f7d09308e9bd252ed2bed3af7afead7a4fd: Status 404 returned error can't find the container with id 3ac160b00f4333cba6787849aeeb9f7d09308e9bd252ed2bed3af7afead7a4fd Jul 06 21:12:59.257660 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:59.257328 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw798\" (UniqueName: \"kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798\") pod \"network-check-target-wjc92\" (UID: \"b4d5a450-11db-44a5-bb22-35f668de29c7\") " pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:12:59.257660 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:59.257642 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs\") pod \"network-metrics-daemon-79rs9\" (UID: \"e4f4e605-6394-4afd-8ef5-55c6fe78cb07\") " pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:12:59.257660 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:59.257531 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 21:12:59.257946 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:59.257695 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 21:12:59.257946 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:59.257714 2581 projected.go:196] Error preparing data for projected volume kube-api-access-pw798 for pod openshift-network-diagnostics/network-check-target-wjc92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 21:12:59.257946 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:59.257763 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 21:12:59.257946 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:59.257777 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798 podName:b4d5a450-11db-44a5-bb22-35f668de29c7 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:01.257755983 +0000 UTC m=+6.020668988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pw798" (UniqueName: "kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798") pod "network-check-target-wjc92" (UID: "b4d5a450-11db-44a5-bb22-35f668de29c7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 21:12:59.257946 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:59.257810 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs podName:e4f4e605-6394-4afd-8ef5-55c6fe78cb07 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:01.257797345 +0000 UTC m=+6.020710331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs") pod "network-metrics-daemon-79rs9" (UID: "e4f4e605-6394-4afd-8ef5-55c6fe78cb07") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 21:12:59.358281 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:59.358133 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret\") pod \"global-pull-secret-syncer-zn2lg\" (UID: \"e4aacd25-63b3-4414-8e53-5d418169367b\") " pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:12:59.358457 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:59.358293 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 21:12:59.358457 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:59.358374 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret podName:e4aacd25-63b3-4414-8e53-5d418169367b nodeName:}" failed. No retries permitted until 2026-07-06 21:13:01.358355841 +0000 UTC m=+6.121268842 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret") pod "global-pull-secret-syncer-zn2lg" (UID: "e4aacd25-63b3-4414-8e53-5d418169367b") : object "kube-system"/"original-pull-secret" not registered Jul 06 21:12:59.685756 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:59.685584 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 21:07:57 +0000 UTC" deadline="2028-03-23 23:58:52.753634316 +0000 UTC" Jul 06 21:12:59.685756 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:59.685645 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15026h45m53.067993252s" Jul 06 21:12:59.785435 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:59.784631 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:12:59.785435 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:59.784759 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zn2lg" podUID="e4aacd25-63b3-4414-8e53-5d418169367b" Jul 06 21:12:59.785435 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:59.784843 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:12:59.785435 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:59.784920 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79rs9" podUID="e4f4e605-6394-4afd-8ef5-55c6fe78cb07" Jul 06 21:12:59.785435 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:59.784969 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:12:59.785435 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:12:59.785033 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjc92" podUID="b4d5a450-11db-44a5-bb22-35f668de29c7" Jul 06 21:12:59.797995 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:59.797870 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" event={"ID":"50de3178-31c9-46a1-9f74-da997e92302a","Type":"ContainerStarted","Data":"f8993003bb829c766c88106dc09c39bc5bfbbb7774e35a341a7b4492339ea2af"} Jul 06 21:12:59.801957 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:59.801914 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" event={"ID":"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713","Type":"ContainerStarted","Data":"cf920acefb47d21bcfee8960c413c1abc93536e05d9a5e771167ebff0d374c88"} Jul 06 21:12:59.813840 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:59.813755 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xngg9" event={"ID":"e27e6563-2401-444f-962e-575b5e9cb93f","Type":"ContainerStarted","Data":"4577e86eb1c123292f8d48ae4061ac3a75d2850a22eeb6a6747b9f5862902108"} Jul 06 21:12:59.835812 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:59.835718 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xq8wc" event={"ID":"0eafbe5d-cd16-448b-b3c0-156ca0c03981","Type":"ContainerStarted","Data":"69c0cb590d4ecdf3f1491f15f1fc30afd9dbea38fb7a4d42a9390d141f380365"} Jul 06 21:12:59.843009 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:59.842918 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lds24" event={"ID":"08f54334-35ee-460d-ae74-75618e3e13c5","Type":"ContainerStarted","Data":"d52fb3ab8ee93e4f3669d60fe271416181871c9d8daa933da67b0a42b7900afc"} Jul 06 21:12:59.849669 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:59.849632 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2q95f" event={"ID":"febe62f7-bf85-42c1-8902-c4dabb9ba553","Type":"ContainerStarted","Data":"4670e09fe7e58829fa761a8d859829fb2b8b50e6cd8a2e22108e6aade1e3985d"} Jul 06 21:12:59.864695 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:59.864183 2581 generic.go:356] "Generic (PLEG): container finished" podID="4316e771519ebb81207605a3b9372f6b" containerID="359bca5b1d980b17d6e29752456e72e1f57312f82c2d3e2b21ace47d61507a29" exitCode=0 Jul 06 21:12:59.864695 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:59.864283 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal" event={"ID":"4316e771519ebb81207605a3b9372f6b","Type":"ContainerDied","Data":"359bca5b1d980b17d6e29752456e72e1f57312f82c2d3e2b21ace47d61507a29"} Jul 06 21:12:59.879304 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:59.879260 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-54.ec2.internal" event={"ID":"b78cd9a8597733097f5408183b6fd42d","Type":"ContainerStarted","Data":"9756f980bb5efca3d985b1547d6e63150a4e6457badc0c66c79d775c825e0a83"} Jul 06 21:12:59.895575 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:59.895537 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x4grf" event={"ID":"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1","Type":"ContainerStarted","Data":"3ac160b00f4333cba6787849aeeb9f7d09308e9bd252ed2bed3af7afead7a4fd"} Jul 06 21:12:59.899551 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:59.899499 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-54.ec2.internal" podStartSLOduration=3.899481834 podStartE2EDuration="3.899481834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:12:59.899205676 +0000 UTC m=+4.662118681" watchObservedRunningTime="2026-07-06 21:12:59.899481834 +0000 UTC m=+4.662394839" Jul 06 21:12:59.906477 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:59.906448 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nqzqb" event={"ID":"75b9c669-e2a1-4c31-b58c-905b30293f0a","Type":"ContainerStarted","Data":"5434b48c106d9af540067cd43cc681185577ba5895d89a6cc31780fbca4e18bc"} Jul 06 21:12:59.927133 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:12:59.927090 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mxfzl" event={"ID":"d38a1184-76ff-4dc4-bacf-bbf90f547ca4","Type":"ContainerStarted","Data":"9ed08dd7010f6cd2e12deeb31e3974845d84b0363410f1234867732ea22f568e"} Jul 06 21:13:00.942283 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:00.942172 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal" event={"ID":"4316e771519ebb81207605a3b9372f6b","Type":"ContainerStarted","Data":"962bb22057892db9cd3a407886ca332c7e22bb9c29a015ed63edfd7bdc098e7a"} Jul 06 21:13:00.957563 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:00.956742 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal" podStartSLOduration=4.95671948 podStartE2EDuration="4.95671948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:00.956595125 +0000 UTC m=+5.719508130" watchObservedRunningTime="2026-07-06 21:13:00.95671948 +0000 UTC m=+5.719632488" Jul 06 21:13:01.274511 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:01.273554 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs\") pod \"network-metrics-daemon-79rs9\" (UID: \"e4f4e605-6394-4afd-8ef5-55c6fe78cb07\") " pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:13:01.274511 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:01.273640 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw798\" (UniqueName: \"kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798\") pod \"network-check-target-wjc92\" (UID: \"b4d5a450-11db-44a5-bb22-35f668de29c7\") " pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:13:01.274511 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:01.273790 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 21:13:01.274511 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:01.273808 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 21:13:01.274511 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:01.273822 2581 projected.go:196] Error preparing data for projected volume kube-api-access-pw798 for pod openshift-network-diagnostics/network-check-target-wjc92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 21:13:01.274511 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:01.273908 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798 podName:b4d5a450-11db-44a5-bb22-35f668de29c7 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:05.273889216 +0000 UTC m=+10.036802212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pw798" (UniqueName: "kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798") pod "network-check-target-wjc92" (UID: "b4d5a450-11db-44a5-bb22-35f668de29c7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 21:13:01.274511 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:01.273983 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 21:13:01.274511 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:01.274015 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs podName:e4f4e605-6394-4afd-8ef5-55c6fe78cb07 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:05.274005796 +0000 UTC m=+10.036918781 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs") pod "network-metrics-daemon-79rs9" (UID: "e4f4e605-6394-4afd-8ef5-55c6fe78cb07") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 21:13:01.375118 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:01.374576 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret\") pod \"global-pull-secret-syncer-zn2lg\" (UID: \"e4aacd25-63b3-4414-8e53-5d418169367b\") " pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:13:01.375118 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:01.374737 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 21:13:01.375118 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:01.374799 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret podName:e4aacd25-63b3-4414-8e53-5d418169367b nodeName:}" failed. No retries permitted until 2026-07-06 21:13:05.374780417 +0000 UTC m=+10.137693409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret") pod "global-pull-secret-syncer-zn2lg" (UID: "e4aacd25-63b3-4414-8e53-5d418169367b") : object "kube-system"/"original-pull-secret" not registered Jul 06 21:13:01.782484 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:01.782452 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:13:01.782651 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:01.782595 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zn2lg" podUID="e4aacd25-63b3-4414-8e53-5d418169367b" Jul 06 21:13:01.782914 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:01.782888 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:13:01.783024 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:01.782995 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79rs9" podUID="e4f4e605-6394-4afd-8ef5-55c6fe78cb07" Jul 06 21:13:01.783097 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:01.783061 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:13:01.783134 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:01.783118 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjc92" podUID="b4d5a450-11db-44a5-bb22-35f668de29c7" Jul 06 21:13:03.784316 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:03.784283 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:13:03.784832 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:03.784394 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:13:03.784832 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:03.784402 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjc92" podUID="b4d5a450-11db-44a5-bb22-35f668de29c7" Jul 06 21:13:03.784832 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:03.784519 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zn2lg" podUID="e4aacd25-63b3-4414-8e53-5d418169367b" Jul 06 21:13:03.784832 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:03.784586 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:13:03.784832 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:03.784667 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79rs9" podUID="e4f4e605-6394-4afd-8ef5-55c6fe78cb07" Jul 06 21:13:05.308514 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:05.308472 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs\") pod \"network-metrics-daemon-79rs9\" (UID: \"e4f4e605-6394-4afd-8ef5-55c6fe78cb07\") " pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:13:05.308949 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:05.308551 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw798\" (UniqueName: \"kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798\") pod \"network-check-target-wjc92\" (UID: \"b4d5a450-11db-44a5-bb22-35f668de29c7\") " pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:13:05.308949 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:05.308691 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 21:13:05.308949 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:05.308708 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 21:13:05.308949 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:05.308720 2581 projected.go:196] Error preparing data for projected volume kube-api-access-pw798 for pod openshift-network-diagnostics/network-check-target-wjc92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 21:13:05.308949 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:05.308776 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798 podName:b4d5a450-11db-44a5-bb22-35f668de29c7 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:13.308758453 +0000 UTC m=+18.071671451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pw798" (UniqueName: "kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798") pod "network-check-target-wjc92" (UID: "b4d5a450-11db-44a5-bb22-35f668de29c7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 21:13:05.309230 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:05.309172 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 21:13:05.309230 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:05.309220 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs podName:e4f4e605-6394-4afd-8ef5-55c6fe78cb07 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:13.309205513 +0000 UTC m=+18.072118544 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs") pod "network-metrics-daemon-79rs9" (UID: "e4f4e605-6394-4afd-8ef5-55c6fe78cb07") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 21:13:05.409408 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:05.409362 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret\") pod \"global-pull-secret-syncer-zn2lg\" (UID: \"e4aacd25-63b3-4414-8e53-5d418169367b\") " pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:13:05.409565 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:05.409537 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 21:13:05.409636 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:05.409613 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret podName:e4aacd25-63b3-4414-8e53-5d418169367b nodeName:}" failed. No retries permitted until 2026-07-06 21:13:13.409590163 +0000 UTC m=+18.172503360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret") pod "global-pull-secret-syncer-zn2lg" (UID: "e4aacd25-63b3-4414-8e53-5d418169367b") : object "kube-system"/"original-pull-secret" not registered Jul 06 21:13:05.784097 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:05.784006 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:13:05.784245 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:05.784137 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79rs9" podUID="e4f4e605-6394-4afd-8ef5-55c6fe78cb07" Jul 06 21:13:05.784521 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:05.784498 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:13:05.784584 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:05.784539 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:13:05.784645 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:05.784630 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjc92" podUID="b4d5a450-11db-44a5-bb22-35f668de29c7" Jul 06 21:13:05.784746 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:05.784719 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zn2lg" podUID="e4aacd25-63b3-4414-8e53-5d418169367b" Jul 06 21:13:07.784944 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:07.784849 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:13:07.784944 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:07.784887 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:13:07.784944 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:07.784892 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:13:07.785579 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:07.784982 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79rs9" podUID="e4f4e605-6394-4afd-8ef5-55c6fe78cb07" Jul 06 21:13:07.785579 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:07.785223 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zn2lg" podUID="e4aacd25-63b3-4414-8e53-5d418169367b" Jul 06 21:13:07.785579 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:07.785374 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjc92" podUID="b4d5a450-11db-44a5-bb22-35f668de29c7" Jul 06 21:13:09.781040 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:09.781004 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:13:09.781525 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:09.781004 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:13:09.781525 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:09.781137 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zn2lg" podUID="e4aacd25-63b3-4414-8e53-5d418169367b" Jul 06 21:13:09.781525 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:09.781170 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:13:09.781525 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:09.781277 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79rs9" podUID="e4f4e605-6394-4afd-8ef5-55c6fe78cb07" Jul 06 21:13:09.781525 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:09.781364 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjc92" podUID="b4d5a450-11db-44a5-bb22-35f668de29c7" Jul 06 21:13:11.780953 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:11.780914 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:13:11.780953 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:11.780950 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:13:11.781391 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:11.780914 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:13:11.781391 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:11.781071 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79rs9" podUID="e4f4e605-6394-4afd-8ef5-55c6fe78cb07" Jul 06 21:13:11.781391 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:11.781125 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zn2lg" podUID="e4aacd25-63b3-4414-8e53-5d418169367b" Jul 06 21:13:11.781391 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:11.781183 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjc92" podUID="b4d5a450-11db-44a5-bb22-35f668de29c7" Jul 06 21:13:13.373739 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:13.373646 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw798\" (UniqueName: \"kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798\") pod \"network-check-target-wjc92\" (UID: \"b4d5a450-11db-44a5-bb22-35f668de29c7\") " pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:13:13.373739 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:13.373706 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs\") pod \"network-metrics-daemon-79rs9\" (UID: \"e4f4e605-6394-4afd-8ef5-55c6fe78cb07\") " pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:13:13.374259 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:13.373811 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 21:13:13.374259 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:13.373822 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 21:13:13.374259 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:13.373838 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 21:13:13.374259 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:13.373850 2581 projected.go:196] Error preparing data for projected volume kube-api-access-pw798 for pod openshift-network-diagnostics/network-check-target-wjc92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 21:13:13.374259 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:13.373876 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs podName:e4f4e605-6394-4afd-8ef5-55c6fe78cb07 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:29.373859147 +0000 UTC m=+34.136772151 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs") pod "network-metrics-daemon-79rs9" (UID: "e4f4e605-6394-4afd-8ef5-55c6fe78cb07") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 21:13:13.374259 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:13.373894 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798 podName:b4d5a450-11db-44a5-bb22-35f668de29c7 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:29.373884582 +0000 UTC m=+34.136797564 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pw798" (UniqueName: "kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798") pod "network-check-target-wjc92" (UID: "b4d5a450-11db-44a5-bb22-35f668de29c7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 21:13:13.474399 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:13.474363 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret\") pod \"global-pull-secret-syncer-zn2lg\" (UID: \"e4aacd25-63b3-4414-8e53-5d418169367b\") " pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:13:13.474576 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:13.474534 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 21:13:13.474649 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:13.474607 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret podName:e4aacd25-63b3-4414-8e53-5d418169367b nodeName:}" failed. No retries permitted until 2026-07-06 21:13:29.474587909 +0000 UTC m=+34.237500891 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret") pod "global-pull-secret-syncer-zn2lg" (UID: "e4aacd25-63b3-4414-8e53-5d418169367b") : object "kube-system"/"original-pull-secret" not registered Jul 06 21:13:13.781007 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:13.780922 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:13:13.781007 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:13.780976 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:13:13.781216 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:13.780925 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:13:13.781216 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:13.781062 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zn2lg" podUID="e4aacd25-63b3-4414-8e53-5d418169367b" Jul 06 21:13:13.781216 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:13.781144 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79rs9" podUID="e4f4e605-6394-4afd-8ef5-55c6fe78cb07" Jul 06 21:13:13.781216 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:13.781209 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjc92" podUID="b4d5a450-11db-44a5-bb22-35f668de29c7" Jul 06 21:13:15.781987 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:15.781951 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:13:15.782475 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:15.782062 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zn2lg" podUID="e4aacd25-63b3-4414-8e53-5d418169367b" Jul 06 21:13:15.782475 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:15.782135 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:13:15.782475 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:15.782235 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79rs9" podUID="e4f4e605-6394-4afd-8ef5-55c6fe78cb07" Jul 06 21:13:15.782475 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:15.782278 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:13:15.782475 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:15.782363 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjc92" podUID="b4d5a450-11db-44a5-bb22-35f668de29c7" Jul 06 21:13:17.780929 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:17.780689 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:13:17.781703 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:17.780689 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:13:17.781703 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:17.780733 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:13:17.781703 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:17.781035 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjc92" podUID="b4d5a450-11db-44a5-bb22-35f668de29c7" Jul 06 21:13:17.781703 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:17.781090 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zn2lg" podUID="e4aacd25-63b3-4414-8e53-5d418169367b" Jul 06 21:13:17.781703 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:17.781169 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79rs9" podUID="e4f4e605-6394-4afd-8ef5-55c6fe78cb07" Jul 06 21:13:17.974798 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:17.974702 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xngg9" event={"ID":"e27e6563-2401-444f-962e-575b5e9cb93f","Type":"ContainerStarted","Data":"987b61225f8f4fd71aaa544e746eba4af4abe56a8102e3daf0ad998c660ca828"} Jul 06 21:13:17.976716 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:17.976685 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xq8wc" event={"ID":"0eafbe5d-cd16-448b-b3c0-156ca0c03981","Type":"ContainerStarted","Data":"ca3bfcb332895c1d32c4aed395584731f393b8ec04ff36613543eedbe67bd6a2"} Jul 06 21:13:17.978687 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:17.978626 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lds24" event={"ID":"08f54334-35ee-460d-ae74-75618e3e13c5","Type":"ContainerStarted","Data":"48640e391e6f9ac6793b994e758461f8464db91a67cd4310774016682f3864c9"} Jul 06 21:13:17.980726 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:17.980688 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2q95f" event={"ID":"febe62f7-bf85-42c1-8902-c4dabb9ba553","Type":"ContainerStarted","Data":"0abcb1c1f1988873573548c8d5527cda572b81fa917b349bd19ee653ce7d7de8"} Jul 06 21:13:17.982477 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:17.982362 2581 generic.go:356] "Generic (PLEG): container finished" podID="b03516a3-66d9-4cd4-b814-dc23dd3bd9a1" containerID="ee04225807d0135a76fc64369ed41b99a5678fd1e1fbbb263ff37ca7865b1d93" exitCode=0 Jul 06 21:13:17.982477 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:17.982392 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x4grf" event={"ID":"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1","Type":"ContainerDied","Data":"ee04225807d0135a76fc64369ed41b99a5678fd1e1fbbb263ff37ca7865b1d93"} Jul 06 21:13:17.983956 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:17.983890 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mxfzl" event={"ID":"d38a1184-76ff-4dc4-bacf-bbf90f547ca4","Type":"ContainerStarted","Data":"c49dc6b403dd509848323704f1f081eab7e39802dca1a0830b321d44ecaa285d"} Jul 06 21:13:17.985665 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:17.985643 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" event={"ID":"50de3178-31c9-46a1-9f74-da997e92302a","Type":"ContainerStarted","Data":"af36b802a8315ba5e646f93d58631ac7ab6d1d46bf4cd7b7fdc29fb14c30c634"} Jul 06 21:13:17.988962 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:17.988937 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" event={"ID":"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713","Type":"ContainerStarted","Data":"9dbd1eb7bf9da043edf1db91036778dfa743523c82a0fdef4482020bb1d4b84a"} Jul 06 21:13:17.989057 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:17.989012 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" event={"ID":"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713","Type":"ContainerStarted","Data":"76797f0c30e0a2d62022aed81f151b4b0cdbbe97893e60cbd7d17776ce04553c"} Jul 06 21:13:17.989127 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:17.989061 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" event={"ID":"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713","Type":"ContainerStarted","Data":"fef41765f00c777f761f9a19ccb63213bf894d2974a38c102b7d9c9d9b630e04"} Jul 06 21:13:17.989127 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:17.989108 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" event={"ID":"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713","Type":"ContainerStarted","Data":"489aa31c9dcd56c3785bfc9bb34a448d8afe6796cf2888562f5cdabe3860c047"} Jul 06 21:13:17.989230 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:17.989162 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" event={"ID":"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713","Type":"ContainerStarted","Data":"0a8bc5d985d68c87308f53ffde2a32a5acaa9750895a5a96b060bac749b56692"} Jul 06 21:13:17.989230 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:17.989209 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" event={"ID":"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713","Type":"ContainerStarted","Data":"5183f8a8fe2e7d1585769c8664a4f23a081d0180416707a8398a17d270976552"} Jul 06 21:13:17.991975 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:17.991925 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xngg9" podStartSLOduration=4.712755558 podStartE2EDuration="22.991909137s" totalImagesPullingTime="18.279153579s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:12:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:17.991312922 +0000 UTC m=+22.754225929" watchObservedRunningTime="2026-07-06 21:13:17.991909137 +0000 UTC m=+22.754822142" Jul 06 21:13:18.039428 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:18.039379 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lds24" podStartSLOduration=3.757320237 podStartE2EDuration="22.039363486s" totalImagesPullingTime="18.282043249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:18.039086966 +0000 UTC m=+22.801999971" watchObservedRunningTime="2026-07-06 21:13:18.039363486 +0000 UTC m=+22.802276489" Jul 06 21:13:18.054983 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:18.054937 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mxfzl" podStartSLOduration=9.07330452 podStartE2EDuration="23.05492389s" totalImagesPullingTime="13.98161937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:12:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:18.054397136 +0000 UTC m=+22.817310142" watchObservedRunningTime="2026-07-06 21:13:18.05492389 +0000 UTC m=+22.817836893" Jul 06 21:13:18.074764 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:18.074706 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xq8wc" podStartSLOduration=4.73384977 podStartE2EDuration="23.074688995s" totalImagesPullingTime="18.340839225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:12:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:18.074583283 +0000 UTC m=+22.837496286" watchObservedRunningTime="2026-07-06 21:13:18.074688995 +0000 UTC m=+22.837602002" Jul 06 21:13:18.094807 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:18.094742 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2q95f" podStartSLOduration=3.820341671 podStartE2EDuration="22.094720152s" totalImagesPullingTime="18.274378481s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:18.094519447 +0000 UTC m=+22.857432451" watchObservedRunningTime="2026-07-06 21:13:18.094720152 +0000 UTC m=+22.857633158" Jul 06 21:13:18.791726 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:18.791701 2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 21:13:18.992964 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:18.992876 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nqzqb" event={"ID":"75b9c669-e2a1-4c31-b58c-905b30293f0a","Type":"ContainerStarted","Data":"e5327e4d36497c584eb700b13bdaa52f1ed40d7573358b953fb083ba882e979d"} Jul 06 21:13:18.994912 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:18.994885 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" event={"ID":"50de3178-31c9-46a1-9f74-da997e92302a","Type":"ContainerStarted","Data":"126d247617e8ffef5866e24695a638b6429062ba755867741049d7eddf88a77d"} Jul 06 21:13:19.008431 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:19.008374 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nqzqb" podStartSLOduration=4.73316488 podStartE2EDuration="23.008358492s" totalImagesPullingTime="18.275193612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:19.007954425 +0000 UTC m=+23.770867430" watchObservedRunningTime="2026-07-06 21:13:19.008358492 +0000 UTC m=+23.771271496" Jul 06 21:13:19.741380 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:19.741252 2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T21:13:18.791720102Z","UUID":"21f241e7-f29a-40a1-a6c5-fa53b1a2b6c9","Handler":null,"Name":"","Endpoint":""} Jul 06 21:13:19.742826 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:19.742803 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 21:13:19.742961 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:19.742844 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 21:13:19.781164 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:19.781125 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:13:19.781332 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:19.781121 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:13:19.781332 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:19.781285 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79rs9" podUID="e4f4e605-6394-4afd-8ef5-55c6fe78cb07" Jul 06 21:13:19.781332 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:19.781316 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:13:19.781500 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:19.781444 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zn2lg" podUID="e4aacd25-63b3-4414-8e53-5d418169367b" Jul 06 21:13:19.781545 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:19.781530 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjc92" podUID="b4d5a450-11db-44a5-bb22-35f668de29c7" Jul 06 21:13:20.578403 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:20.578125 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mxfzl" Jul 06 21:13:20.578805 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:20.578788 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mxfzl" Jul 06 21:13:21.000455 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:21.000371 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" event={"ID":"50de3178-31c9-46a1-9f74-da997e92302a","Type":"ContainerStarted","Data":"bfa5c7c335487b2897df90ff9f6c306d85672b55a4aaceddd5314b2a6dc126a5"} Jul 06 21:13:21.003592 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:21.003570 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" event={"ID":"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713","Type":"ContainerStarted","Data":"c6ac4678d14c7e3db6c71a685dc67d5f53594084bc91cdf6620e4403e88e14fa"} Jul 06 21:13:21.003719 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:21.003630 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mxfzl" Jul 06 21:13:21.004131 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:21.004108 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mxfzl" Jul 06 21:13:21.019423 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:21.019387 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvtcl" podStartSLOduration=4.1308548609999995 podStartE2EDuration="25.019374513s" totalImagesPullingTime="20.888519652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:12:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:21.018867206 +0000 UTC m=+25.781780209" watchObservedRunningTime="2026-07-06 21:13:21.019374513 +0000 UTC m=+25.782287518" Jul 06 21:13:21.781351 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:21.781308 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:13:21.781795 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:21.781309 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:13:21.781795 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:21.781464 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjc92" podUID="b4d5a450-11db-44a5-bb22-35f668de29c7" Jul 06 21:13:21.781795 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:21.781323 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:13:21.781795 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:21.781551 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zn2lg" podUID="e4aacd25-63b3-4414-8e53-5d418169367b" Jul 06 21:13:21.781795 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:21.781609 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79rs9" podUID="e4f4e605-6394-4afd-8ef5-55c6fe78cb07" Jul 06 21:13:23.781072 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:23.781035 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:13:23.781827 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:23.781039 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:13:23.781827 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:23.781134 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zn2lg" podUID="e4aacd25-63b3-4414-8e53-5d418169367b" Jul 06 21:13:23.781827 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:23.781162 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:13:23.781827 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:23.781249 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79rs9" podUID="e4f4e605-6394-4afd-8ef5-55c6fe78cb07" Jul 06 21:13:23.781827 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:23.781323 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjc92" podUID="b4d5a450-11db-44a5-bb22-35f668de29c7" Jul 06 21:13:24.013372 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:24.013199 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" event={"ID":"a276f91f-2f3c-4f2c-85ff-f53d4bc6c713","Type":"ContainerStarted","Data":"476d9997f43d6804f6d3ddfc525d30721c2839b02f76a84eebcb14dfe49363a9"} Jul 06 21:13:24.013540 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:24.013415 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:13:24.013540 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:24.013459 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:13:24.029105 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:24.029081 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:13:24.040943 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:24.040837 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" podStartSLOduration=9.694082116 podStartE2EDuration="28.040822704s" totalImagesPullingTime="18.346740588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:24.039543046 +0000 UTC m=+28.802456049" watchObservedRunningTime="2026-07-06 21:13:24.040822704 +0000 UTC m=+28.803735708" Jul 06 21:13:25.017272 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:25.017240 2581 generic.go:356] "Generic (PLEG): container finished" podID="b03516a3-66d9-4cd4-b814-dc23dd3bd9a1" containerID="852ece2d48cd041c3febd2fe26ff154885fa8993f4d11e69ef0814ed93ff0df2" exitCode=0 Jul 06 21:13:25.018048 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:25.017327 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x4grf" event={"ID":"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1","Type":"ContainerDied","Data":"852ece2d48cd041c3febd2fe26ff154885fa8993f4d11e69ef0814ed93ff0df2"} Jul 06 21:13:25.018048 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:25.017687 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:13:25.035402 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:25.033225 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:13:25.782187 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:25.782026 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:13:25.782379 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:25.782094 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:13:25.782379 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:25.782114 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:13:25.782503 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:25.782375 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79rs9" podUID="e4f4e605-6394-4afd-8ef5-55c6fe78cb07" Jul 06 21:13:25.782503 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:25.782246 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zn2lg" podUID="e4aacd25-63b3-4414-8e53-5d418169367b" Jul 06 21:13:25.782503 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:25.782482 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjc92" podUID="b4d5a450-11db-44a5-bb22-35f668de29c7" Jul 06 21:13:25.857730 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:25.857698 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zn2lg"] Jul 06 21:13:25.858243 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:25.858223 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-79rs9"] Jul 06 21:13:25.859834 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:25.859789 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wjc92"] Jul 06 21:13:26.019568 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:26.019487 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:13:26.019568 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:26.019493 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:13:26.019568 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:26.019496 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:13:26.020130 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:26.019594 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79rs9" podUID="e4f4e605-6394-4afd-8ef5-55c6fe78cb07" Jul 06 21:13:26.020130 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:26.019649 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zn2lg" podUID="e4aacd25-63b3-4414-8e53-5d418169367b" Jul 06 21:13:26.020130 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:26.020124 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjc92" podUID="b4d5a450-11db-44a5-bb22-35f668de29c7" Jul 06 21:13:27.030739 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:27.030702 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x4grf" event={"ID":"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1","Type":"ContainerDied","Data":"bb1c5f761d88837111aa70a3ff15beca11de56848fbe21684326536049315fe1"} Jul 06 21:13:27.031548 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:27.030723 2581 generic.go:356] "Generic (PLEG): container finished" podID="b03516a3-66d9-4cd4-b814-dc23dd3bd9a1" containerID="bb1c5f761d88837111aa70a3ff15beca11de56848fbe21684326536049315fe1" exitCode=0 Jul 06 21:13:27.780754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:27.780722 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:13:27.780941 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:27.780721 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:13:27.780941 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:27.780852 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjc92" podUID="b4d5a450-11db-44a5-bb22-35f668de29c7" Jul 06 21:13:27.780941 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:27.780901 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:13:27.781058 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:27.780974 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zn2lg" podUID="e4aacd25-63b3-4414-8e53-5d418169367b" Jul 06 21:13:27.781103 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:27.781054 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79rs9" podUID="e4f4e605-6394-4afd-8ef5-55c6fe78cb07" Jul 06 21:13:29.038032 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:29.037945 2581 generic.go:356] "Generic (PLEG): container finished" podID="b03516a3-66d9-4cd4-b814-dc23dd3bd9a1" containerID="68b5d428e91a798cf70fa7bcc44d4eb1edc4c3c1bb92743b24b0e953393e7373" exitCode=0 Jul 06 21:13:29.038032 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:29.037997 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x4grf" event={"ID":"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1","Type":"ContainerDied","Data":"68b5d428e91a798cf70fa7bcc44d4eb1edc4c3c1bb92743b24b0e953393e7373"} Jul 06 21:13:29.385624 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:29.385544 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw798\" (UniqueName: \"kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798\") pod \"network-check-target-wjc92\" (UID: \"b4d5a450-11db-44a5-bb22-35f668de29c7\") " pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:13:29.385624 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:29.385593 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs\") pod \"network-metrics-daemon-79rs9\" (UID: \"e4f4e605-6394-4afd-8ef5-55c6fe78cb07\") " pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:13:29.385792 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:29.385701 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 21:13:29.385792 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:29.385717 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 21:13:29.385792 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:29.385737 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 21:13:29.385792 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:29.385748 2581 projected.go:196] Error preparing data for projected volume kube-api-access-pw798 for pod openshift-network-diagnostics/network-check-target-wjc92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 21:13:29.385792 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:29.385756 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs podName:e4f4e605-6394-4afd-8ef5-55c6fe78cb07 nodeName:}" failed. No retries permitted until 2026-07-06 21:14:01.385738786 +0000 UTC m=+66.148651786 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs") pod "network-metrics-daemon-79rs9" (UID: "e4f4e605-6394-4afd-8ef5-55c6fe78cb07") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 21:13:29.385792 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:29.385793 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798 podName:b4d5a450-11db-44a5-bb22-35f668de29c7 nodeName:}" failed. No retries permitted until 2026-07-06 21:14:01.385782477 +0000 UTC m=+66.148695463 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pw798" (UniqueName: "kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798") pod "network-check-target-wjc92" (UID: "b4d5a450-11db-44a5-bb22-35f668de29c7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 21:13:29.485917 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:29.485880 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret\") pod \"global-pull-secret-syncer-zn2lg\" (UID: \"e4aacd25-63b3-4414-8e53-5d418169367b\") " pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:13:29.486087 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:29.485997 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 21:13:29.486087 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:29.486049 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret podName:e4aacd25-63b3-4414-8e53-5d418169367b nodeName:}" failed. No retries permitted until 2026-07-06 21:14:01.486036431 +0000 UTC m=+66.248949417 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret") pod "global-pull-secret-syncer-zn2lg" (UID: "e4aacd25-63b3-4414-8e53-5d418169367b") : object "kube-system"/"original-pull-secret" not registered Jul 06 21:13:29.781033 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:29.780954 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:13:29.781033 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:29.780989 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:13:29.781233 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:29.780954 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:13:29.781233 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:29.781086 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79rs9" podUID="e4f4e605-6394-4afd-8ef5-55c6fe78cb07" Jul 06 21:13:29.781233 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:29.781150 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zn2lg" podUID="e4aacd25-63b3-4414-8e53-5d418169367b" Jul 06 21:13:29.781233 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:29.781211 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjc92" podUID="b4d5a450-11db-44a5-bb22-35f668de29c7" Jul 06 21:13:30.961584 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:30.961511 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeReady" Jul 06 21:13:30.961962 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:30.961682 2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 21:13:31.018017 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.017987 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jxz5h"] Jul 06 21:13:31.019916 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.019898 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sstxl"] Jul 06 21:13:31.020085 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.020060 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jxz5h" Jul 06 21:13:31.021882 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.021865 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sstxl" Jul 06 21:13:31.023087 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.022859 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 21:13:31.023200 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.023100 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 21:13:31.023200 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.023112 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 21:13:31.023200 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.023128 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kccmm\"" Jul 06 21:13:31.024546 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.024526 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 21:13:31.024820 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.024804 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4w8fd\"" Jul 06 21:13:31.024922 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.024908 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 21:13:31.034490 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.034445 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jxz5h"] Jul 06 21:13:31.050576 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.050519 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sstxl"] Jul 06 21:13:31.100354 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.100304 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6hzz\" (UniqueName: \"kubernetes.io/projected/704cb9c1-1b35-419a-90f0-0cf0dd5498f5-kube-api-access-p6hzz\") pod \"ingress-canary-jxz5h\" (UID: \"704cb9c1-1b35-419a-90f0-0cf0dd5498f5\") " pod="openshift-ingress-canary/ingress-canary-jxz5h" Jul 06 21:13:31.100516 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.100414 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c160ad01-8d36-4567-945d-fd6810e76537-metrics-tls\") pod \"dns-default-sstxl\" (UID: \"c160ad01-8d36-4567-945d-fd6810e76537\") " pod="openshift-dns/dns-default-sstxl" Jul 06 21:13:31.100516 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.100452 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/704cb9c1-1b35-419a-90f0-0cf0dd5498f5-cert\") pod \"ingress-canary-jxz5h\" (UID: \"704cb9c1-1b35-419a-90f0-0cf0dd5498f5\") " pod="openshift-ingress-canary/ingress-canary-jxz5h" Jul 06 21:13:31.100641 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.100550 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nfh9\" (UniqueName: \"kubernetes.io/projected/c160ad01-8d36-4567-945d-fd6810e76537-kube-api-access-9nfh9\") pod \"dns-default-sstxl\" (UID: \"c160ad01-8d36-4567-945d-fd6810e76537\") " pod="openshift-dns/dns-default-sstxl" Jul 06 21:13:31.100641 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.100635 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c160ad01-8d36-4567-945d-fd6810e76537-tmp-dir\") pod \"dns-default-sstxl\" (UID: \"c160ad01-8d36-4567-945d-fd6810e76537\") " pod="openshift-dns/dns-default-sstxl" Jul 06 21:13:31.100723 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.100690 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c160ad01-8d36-4567-945d-fd6810e76537-config-volume\") pod \"dns-default-sstxl\" (UID: \"c160ad01-8d36-4567-945d-fd6810e76537\") " pod="openshift-dns/dns-default-sstxl" Jul 06 21:13:31.201511 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.201475 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c160ad01-8d36-4567-945d-fd6810e76537-tmp-dir\") pod \"dns-default-sstxl\" (UID: \"c160ad01-8d36-4567-945d-fd6810e76537\") " pod="openshift-dns/dns-default-sstxl" Jul 06 21:13:31.201511 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.201524 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c160ad01-8d36-4567-945d-fd6810e76537-config-volume\") pod \"dns-default-sstxl\" (UID: \"c160ad01-8d36-4567-945d-fd6810e76537\") " pod="openshift-dns/dns-default-sstxl" Jul 06 21:13:31.201742 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.201570 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6hzz\" (UniqueName: \"kubernetes.io/projected/704cb9c1-1b35-419a-90f0-0cf0dd5498f5-kube-api-access-p6hzz\") pod \"ingress-canary-jxz5h\" (UID: \"704cb9c1-1b35-419a-90f0-0cf0dd5498f5\") " pod="openshift-ingress-canary/ingress-canary-jxz5h" Jul 06 21:13:31.201742 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.201602 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c160ad01-8d36-4567-945d-fd6810e76537-metrics-tls\") pod \"dns-default-sstxl\" (UID: \"c160ad01-8d36-4567-945d-fd6810e76537\") " pod="openshift-dns/dns-default-sstxl" Jul 06 21:13:31.201742 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.201629 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/704cb9c1-1b35-419a-90f0-0cf0dd5498f5-cert\") pod \"ingress-canary-jxz5h\" (UID: \"704cb9c1-1b35-419a-90f0-0cf0dd5498f5\") " pod="openshift-ingress-canary/ingress-canary-jxz5h" Jul 06 21:13:31.201742 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.201677 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nfh9\" (UniqueName: \"kubernetes.io/projected/c160ad01-8d36-4567-945d-fd6810e76537-kube-api-access-9nfh9\") pod \"dns-default-sstxl\" (UID: \"c160ad01-8d36-4567-945d-fd6810e76537\") " pod="openshift-dns/dns-default-sstxl" Jul 06 21:13:31.201742 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:31.201727 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 21:13:31.201977 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:31.201789 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c160ad01-8d36-4567-945d-fd6810e76537-metrics-tls podName:c160ad01-8d36-4567-945d-fd6810e76537 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:31.701769186 +0000 UTC m=+36.464682182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c160ad01-8d36-4567-945d-fd6810e76537-metrics-tls") pod "dns-default-sstxl" (UID: "c160ad01-8d36-4567-945d-fd6810e76537") : secret "dns-default-metrics-tls" not found Jul 06 21:13:31.201977 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.201887 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c160ad01-8d36-4567-945d-fd6810e76537-tmp-dir\") pod \"dns-default-sstxl\" (UID: \"c160ad01-8d36-4567-945d-fd6810e76537\") " pod="openshift-dns/dns-default-sstxl" Jul 06 21:13:31.202245 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.202214 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c160ad01-8d36-4567-945d-fd6810e76537-config-volume\") pod \"dns-default-sstxl\" (UID: \"c160ad01-8d36-4567-945d-fd6810e76537\") " pod="openshift-dns/dns-default-sstxl" Jul 06 21:13:31.206410 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.206382 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/704cb9c1-1b35-419a-90f0-0cf0dd5498f5-cert\") pod \"ingress-canary-jxz5h\" (UID: \"704cb9c1-1b35-419a-90f0-0cf0dd5498f5\") " pod="openshift-ingress-canary/ingress-canary-jxz5h" Jul 06 21:13:31.210814 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.210790 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nfh9\" (UniqueName: \"kubernetes.io/projected/c160ad01-8d36-4567-945d-fd6810e76537-kube-api-access-9nfh9\") pod \"dns-default-sstxl\" (UID: \"c160ad01-8d36-4567-945d-fd6810e76537\") " pod="openshift-dns/dns-default-sstxl" Jul 06 21:13:31.210814 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.210807 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6hzz\" (UniqueName: \"kubernetes.io/projected/704cb9c1-1b35-419a-90f0-0cf0dd5498f5-kube-api-access-p6hzz\") pod \"ingress-canary-jxz5h\" (UID: \"704cb9c1-1b35-419a-90f0-0cf0dd5498f5\") " pod="openshift-ingress-canary/ingress-canary-jxz5h" Jul 06 21:13:31.332157 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.332042 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jxz5h" Jul 06 21:13:31.500068 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.499927 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jxz5h"] Jul 06 21:13:31.504589 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:13:31.504560 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704cb9c1_1b35_419a_90f0_0cf0dd5498f5.slice/crio-32db17f06331fd9544e7cf549a484bc97bac8a478f2b8f0c75e9edc9cf11837a WatchSource:0}: Error finding container 32db17f06331fd9544e7cf549a484bc97bac8a478f2b8f0c75e9edc9cf11837a: Status 404 returned error can't find the container with id 32db17f06331fd9544e7cf549a484bc97bac8a478f2b8f0c75e9edc9cf11837a Jul 06 21:13:31.705208 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.705109 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c160ad01-8d36-4567-945d-fd6810e76537-metrics-tls\") pod \"dns-default-sstxl\" (UID: \"c160ad01-8d36-4567-945d-fd6810e76537\") " pod="openshift-dns/dns-default-sstxl" Jul 06 21:13:31.705397 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:31.705254 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 21:13:31.705397 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:31.705364 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c160ad01-8d36-4567-945d-fd6810e76537-metrics-tls podName:c160ad01-8d36-4567-945d-fd6810e76537 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:32.705324719 +0000 UTC m=+37.468237714 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c160ad01-8d36-4567-945d-fd6810e76537-metrics-tls") pod "dns-default-sstxl" (UID: "c160ad01-8d36-4567-945d-fd6810e76537") : secret "dns-default-metrics-tls" not found Jul 06 21:13:31.780892 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.780859 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:13:31.781077 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.780859 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:13:31.781226 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.781198 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:13:31.784752 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.784701 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 21:13:31.784752 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.784701 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c2dbc\"" Jul 06 21:13:31.784752 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.784737 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 21:13:31.784752 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.784701 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kz87d\"" Jul 06 21:13:31.785125 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.784973 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 21:13:31.785213 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:31.785145 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 21:13:32.046968 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:32.046873 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jxz5h" event={"ID":"704cb9c1-1b35-419a-90f0-0cf0dd5498f5","Type":"ContainerStarted","Data":"32db17f06331fd9544e7cf549a484bc97bac8a478f2b8f0c75e9edc9cf11837a"} Jul 06 21:13:32.711535 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:32.711497 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c160ad01-8d36-4567-945d-fd6810e76537-metrics-tls\") pod \"dns-default-sstxl\" (UID: \"c160ad01-8d36-4567-945d-fd6810e76537\") " pod="openshift-dns/dns-default-sstxl" Jul 06 21:13:32.711726 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:32.711703 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 21:13:32.711817 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:32.711806 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c160ad01-8d36-4567-945d-fd6810e76537-metrics-tls podName:c160ad01-8d36-4567-945d-fd6810e76537 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:34.711763957 +0000 UTC m=+39.474676944 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c160ad01-8d36-4567-945d-fd6810e76537-metrics-tls") pod "dns-default-sstxl" (UID: "c160ad01-8d36-4567-945d-fd6810e76537") : secret "dns-default-metrics-tls" not found Jul 06 21:13:32.974016 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:32.973932 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2tsvl"] Jul 06 21:13:32.977165 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:32.977141 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2tsvl" Jul 06 21:13:32.980099 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:32.980075 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 06 21:13:32.980226 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:32.980075 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 06 21:13:32.980295 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:32.980225 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-tlkvk\"" Jul 06 21:13:32.980367 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:32.980319 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 06 21:13:32.981167 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:32.981143 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 06 21:13:32.985066 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:32.985036 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-2m2h6"] Jul 06 21:13:32.987125 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:32.987106 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85bbcdb746-lph2c"] Jul 06 21:13:32.987294 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:32.987235 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2m2h6" Jul 06 21:13:32.989153 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:32.989135 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:32.990016 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:32.989954 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9l75j\"" Jul 06 21:13:32.992253 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:32.992231 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dgf9v\"" Jul 06 21:13:32.992848 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:32.992455 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 21:13:32.992848 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:32.992510 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2tsvl"] Jul 06 21:13:32.992848 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:32.992675 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 21:13:32.993030 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:32.992870 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 21:13:32.997493 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:32.997471 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 21:13:32.999798 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:32.999778 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-2m2h6"] Jul 06 21:13:33.006555 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.006532 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85bbcdb746-lph2c"] Jul 06 21:13:33.094423 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.093267 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9qb5m"] Jul 06 21:13:33.095725 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.095699 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-cnsh7"] Jul 06 21:13:33.095851 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.095825 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9qb5m" Jul 06 21:13:33.097499 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.097477 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-629h4"] Jul 06 21:13:33.098388 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.098363 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" Jul 06 21:13:33.098702 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.098683 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-85zj9\"" Jul 06 21:13:33.098922 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.098900 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 06 21:13:33.099025 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.098922 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 06 21:13:33.099086 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.099055 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 06 21:13:33.099498 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.099479 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-629h4" Jul 06 21:13:33.100962 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.100796 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 06 21:13:33.100962 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.100886 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 06 21:13:33.101141 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.101119 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8rchs\"" Jul 06 21:13:33.101546 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.101528 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 06 21:13:33.103179 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.102517 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 06 21:13:33.103179 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.102541 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 21:13:33.103179 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.102798 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 06 21:13:33.103179 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.102828 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 21:13:33.106966 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.106389 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9qb5m"] Jul 06 21:13:33.106966 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.106667 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-gmpw5\"" Jul 06 21:13:33.111184 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.111128 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-cnsh7"] Jul 06 21:13:33.113375 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.111831 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-629h4"] Jul 06 21:13:33.113841 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.113725 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da180245-4bba-4cbd-8bb6-6e370897c52d-ca-trust-extracted\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.113841 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.113808 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da180245-4bba-4cbd-8bb6-6e370897c52d-installation-pull-secrets\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.113973 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.113872 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-2tsvl\" (UID: \"b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2tsvl" Jul 06 21:13:33.113973 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.113945 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-certificates\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.114127 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.113977 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56kbh\" (UniqueName: \"kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-kube-api-access-56kbh\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.114127 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.114039 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.114226 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.114123 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da180245-4bba-4cbd-8bb6-6e370897c52d-trusted-ca\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.114226 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.114191 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-bound-sa-token\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.114226 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.114217 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6lfg\" (UniqueName: \"kubernetes.io/projected/b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be-kube-api-access-z6lfg\") pod \"kube-storage-version-migrator-operator-846bf668f7-2tsvl\" (UID: \"b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2tsvl" Jul 06 21:13:33.114394 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.114265 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btcf9\" (UniqueName: \"kubernetes.io/projected/a0b9e0da-5047-4951-8240-59dfad0b3efa-kube-api-access-btcf9\") pod \"network-check-source-989f8b8dc-2m2h6\" (UID: \"a0b9e0da-5047-4951-8240-59dfad0b3efa\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2m2h6" Jul 06 21:13:33.114394 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.114292 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da180245-4bba-4cbd-8bb6-6e370897c52d-image-registry-private-configuration\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.114394 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.114324 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-2tsvl\" (UID: \"b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2tsvl" Jul 06 21:13:33.126991 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.126964 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 06 21:13:33.174185 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.174159 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-548d5d67c5-5qdmw"] Jul 06 21:13:33.176401 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.176381 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:33.178826 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.178808 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 06 21:13:33.179955 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.179934 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 06 21:13:33.180036 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.179996 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 21:13:33.180147 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.179936 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9kv9l\"" Jul 06 21:13:33.180189 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.180144 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 21:13:33.180410 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.180394 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 06 21:13:33.180602 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.180587 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 06 21:13:33.200159 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.200131 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-548d5d67c5-5qdmw"] Jul 06 21:13:33.214777 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.214738 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-2tsvl\" (UID: \"b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2tsvl" Jul 06 21:13:33.214933 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.214788 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da180245-4bba-4cbd-8bb6-6e370897c52d-ca-trust-extracted\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.214933 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.214832 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a156ee3-56b6-45e1-80de-debe10e30cca-serving-cert\") pod \"insights-operator-5c847dcfc4-cnsh7\" (UID: \"2a156ee3-56b6-45e1-80de-debe10e30cca\") " pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" Jul 06 21:13:33.214933 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.214872 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2a156ee3-56b6-45e1-80de-debe10e30cca-snapshots\") pod \"insights-operator-5c847dcfc4-cnsh7\" (UID: \"2a156ee3-56b6-45e1-80de-debe10e30cca\") " pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" Jul 06 21:13:33.214933 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.214901 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a156ee3-56b6-45e1-80de-debe10e30cca-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-cnsh7\" (UID: \"2a156ee3-56b6-45e1-80de-debe10e30cca\") " pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" Jul 06 21:13:33.215149 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.214937 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.215149 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.214961 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29wvt\" (UniqueName: \"kubernetes.io/projected/2a156ee3-56b6-45e1-80de-debe10e30cca-kube-api-access-29wvt\") pod \"insights-operator-5c847dcfc4-cnsh7\" (UID: \"2a156ee3-56b6-45e1-80de-debe10e30cca\") " pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" Jul 06 21:13:33.215149 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.215090 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/786193aa-77dd-42c4-8944-0aadc3a864f0-config\") pod \"service-ca-operator-5d46d9ffd4-9qb5m\" (UID: \"786193aa-77dd-42c4-8944-0aadc3a864f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9qb5m" Jul 06 21:13:33.215149 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.215123 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a156ee3-56b6-45e1-80de-debe10e30cca-tmp\") pod \"insights-operator-5c847dcfc4-cnsh7\" (UID: \"2a156ee3-56b6-45e1-80de-debe10e30cca\") " pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" Jul 06 21:13:33.215386 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.215157 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da180245-4bba-4cbd-8bb6-6e370897c52d-installation-pull-secrets\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.215386 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.215177 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-2tsvl\" (UID: \"b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2tsvl" Jul 06 21:13:33.215386 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.215196 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlhlj\" (UniqueName: \"kubernetes.io/projected/68f8fa10-1463-4367-be99-23850541f8fc-kube-api-access-nlhlj\") pod \"volume-data-source-validator-5d456fdbcd-629h4\" (UID: \"68f8fa10-1463-4367-be99-23850541f8fc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-629h4" Jul 06 21:13:33.215386 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.215214 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/786193aa-77dd-42c4-8944-0aadc3a864f0-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-9qb5m\" (UID: \"786193aa-77dd-42c4-8944-0aadc3a864f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9qb5m" Jul 06 21:13:33.215386 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.215235 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-certificates\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.215386 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.215261 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56kbh\" (UniqueName: \"kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-kube-api-access-56kbh\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.215386 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.215316 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da180245-4bba-4cbd-8bb6-6e370897c52d-ca-trust-extracted\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.215807 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:33.215736 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 21:13:33.215807 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:33.215755 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85bbcdb746-lph2c: secret "image-registry-tls" not found Jul 06 21:13:33.215906 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:33.215824 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls podName:da180245-4bba-4cbd-8bb6-6e370897c52d nodeName:}" failed. No retries permitted until 2026-07-06 21:13:33.715804976 +0000 UTC m=+38.478717974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls") pod "image-registry-85bbcdb746-lph2c" (UID: "da180245-4bba-4cbd-8bb6-6e370897c52d") : secret "image-registry-tls" not found Jul 06 21:13:33.216046 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.215737 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-2tsvl\" (UID: \"b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2tsvl" Jul 06 21:13:33.216133 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.216088 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da180245-4bba-4cbd-8bb6-6e370897c52d-trusted-ca\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.216133 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.216124 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-bound-sa-token\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.216232 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.216205 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-certificates\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.216477 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.216449 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8d7g\" (UniqueName: \"kubernetes.io/projected/786193aa-77dd-42c4-8944-0aadc3a864f0-kube-api-access-r8d7g\") pod \"service-ca-operator-5d46d9ffd4-9qb5m\" (UID: \"786193aa-77dd-42c4-8944-0aadc3a864f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9qb5m" Jul 06 21:13:33.216581 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.216527 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6lfg\" (UniqueName: \"kubernetes.io/projected/b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be-kube-api-access-z6lfg\") pod \"kube-storage-version-migrator-operator-846bf668f7-2tsvl\" (UID: \"b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2tsvl" Jul 06 21:13:33.216800 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.216771 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btcf9\" (UniqueName: \"kubernetes.io/projected/a0b9e0da-5047-4951-8240-59dfad0b3efa-kube-api-access-btcf9\") pod \"network-check-source-989f8b8dc-2m2h6\" (UID: \"a0b9e0da-5047-4951-8240-59dfad0b3efa\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2m2h6" Jul 06 21:13:33.216897 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.216808 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da180245-4bba-4cbd-8bb6-6e370897c52d-image-registry-private-configuration\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.216897 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.216836 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a156ee3-56b6-45e1-80de-debe10e30cca-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-cnsh7\" (UID: \"2a156ee3-56b6-45e1-80de-debe10e30cca\") " pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" Jul 06 21:13:33.217011 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.216957 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da180245-4bba-4cbd-8bb6-6e370897c52d-trusted-ca\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.220102 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.220074 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da180245-4bba-4cbd-8bb6-6e370897c52d-image-registry-private-configuration\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.220201 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.220104 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da180245-4bba-4cbd-8bb6-6e370897c52d-installation-pull-secrets\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.220272 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.220263 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-2tsvl\" (UID: \"b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2tsvl" Jul 06 21:13:33.242127 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.242052 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btcf9\" (UniqueName: \"kubernetes.io/projected/a0b9e0da-5047-4951-8240-59dfad0b3efa-kube-api-access-btcf9\") pod \"network-check-source-989f8b8dc-2m2h6\" (UID: \"a0b9e0da-5047-4951-8240-59dfad0b3efa\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2m2h6" Jul 06 21:13:33.242263 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.242166 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-bound-sa-token\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.242263 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.242248 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56kbh\" (UniqueName: \"kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-kube-api-access-56kbh\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.243983 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.243958 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6lfg\" (UniqueName: \"kubernetes.io/projected/b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be-kube-api-access-z6lfg\") pod \"kube-storage-version-migrator-operator-846bf668f7-2tsvl\" (UID: \"b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2tsvl" Jul 06 21:13:33.290145 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.290108 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2tsvl" Jul 06 21:13:33.303319 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.303287 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2m2h6" Jul 06 21:13:33.318132 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.318099 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29wvt\" (UniqueName: \"kubernetes.io/projected/2a156ee3-56b6-45e1-80de-debe10e30cca-kube-api-access-29wvt\") pod \"insights-operator-5c847dcfc4-cnsh7\" (UID: \"2a156ee3-56b6-45e1-80de-debe10e30cca\") " pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" Jul 06 21:13:33.318292 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.318172 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgqxl\" (UniqueName: \"kubernetes.io/projected/ad697c3c-f1a2-4209-827e-76d6164bd893-kube-api-access-mgqxl\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:33.318292 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.318210 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/786193aa-77dd-42c4-8944-0aadc3a864f0-config\") pod \"service-ca-operator-5d46d9ffd4-9qb5m\" (UID: \"786193aa-77dd-42c4-8944-0aadc3a864f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9qb5m" Jul 06 21:13:33.318292 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.318235 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a156ee3-56b6-45e1-80de-debe10e30cca-tmp\") pod \"insights-operator-5c847dcfc4-cnsh7\" (UID: \"2a156ee3-56b6-45e1-80de-debe10e30cca\") " pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" Jul 06 21:13:33.318292 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.318277 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlhlj\" (UniqueName: \"kubernetes.io/projected/68f8fa10-1463-4367-be99-23850541f8fc-kube-api-access-nlhlj\") pod \"volume-data-source-validator-5d456fdbcd-629h4\" (UID: \"68f8fa10-1463-4367-be99-23850541f8fc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-629h4" Jul 06 21:13:33.318527 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.318307 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/786193aa-77dd-42c4-8944-0aadc3a864f0-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-9qb5m\" (UID: \"786193aa-77dd-42c4-8944-0aadc3a864f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9qb5m" Jul 06 21:13:33.318527 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.318362 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-default-certificate\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:33.318527 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.318402 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8d7g\" (UniqueName: \"kubernetes.io/projected/786193aa-77dd-42c4-8944-0aadc3a864f0-kube-api-access-r8d7g\") pod \"service-ca-operator-5d46d9ffd4-9qb5m\" (UID: \"786193aa-77dd-42c4-8944-0aadc3a864f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9qb5m" Jul 06 21:13:33.318527 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.318445 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-stats-auth\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:33.318527 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.318476 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a156ee3-56b6-45e1-80de-debe10e30cca-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-cnsh7\" (UID: \"2a156ee3-56b6-45e1-80de-debe10e30cca\") " pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" Jul 06 21:13:33.318761 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.318631 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad697c3c-f1a2-4209-827e-76d6164bd893-service-ca-bundle\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:33.318761 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.318694 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a156ee3-56b6-45e1-80de-debe10e30cca-serving-cert\") pod \"insights-operator-5c847dcfc4-cnsh7\" (UID: \"2a156ee3-56b6-45e1-80de-debe10e30cca\") " pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" Jul 06 21:13:33.318761 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.318721 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-metrics-certs\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:33.318896 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.318763 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2a156ee3-56b6-45e1-80de-debe10e30cca-snapshots\") pod \"insights-operator-5c847dcfc4-cnsh7\" (UID: \"2a156ee3-56b6-45e1-80de-debe10e30cca\") " pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" Jul 06 21:13:33.318896 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.318794 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a156ee3-56b6-45e1-80de-debe10e30cca-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-cnsh7\" (UID: \"2a156ee3-56b6-45e1-80de-debe10e30cca\") " pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" Jul 06 21:13:33.318896 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.318823 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a156ee3-56b6-45e1-80de-debe10e30cca-tmp\") pod \"insights-operator-5c847dcfc4-cnsh7\" (UID: \"2a156ee3-56b6-45e1-80de-debe10e30cca\") " pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" Jul 06 21:13:33.319420 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.319398 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/786193aa-77dd-42c4-8944-0aadc3a864f0-config\") pod \"service-ca-operator-5d46d9ffd4-9qb5m\" (UID: \"786193aa-77dd-42c4-8944-0aadc3a864f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9qb5m" Jul 06 21:13:33.319559 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.319487 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a156ee3-56b6-45e1-80de-debe10e30cca-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-cnsh7\" (UID: \"2a156ee3-56b6-45e1-80de-debe10e30cca\") " pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" Jul 06 21:13:33.319765 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.319718 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2a156ee3-56b6-45e1-80de-debe10e30cca-snapshots\") pod \"insights-operator-5c847dcfc4-cnsh7\" (UID: \"2a156ee3-56b6-45e1-80de-debe10e30cca\") " pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" Jul 06 21:13:33.320297 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.320275 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a156ee3-56b6-45e1-80de-debe10e30cca-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-cnsh7\" (UID: \"2a156ee3-56b6-45e1-80de-debe10e30cca\") " pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" Jul 06 21:13:33.322085 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.322061 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/786193aa-77dd-42c4-8944-0aadc3a864f0-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-9qb5m\" (UID: \"786193aa-77dd-42c4-8944-0aadc3a864f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9qb5m" Jul 06 21:13:33.322220 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.322196 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a156ee3-56b6-45e1-80de-debe10e30cca-serving-cert\") pod \"insights-operator-5c847dcfc4-cnsh7\" (UID: \"2a156ee3-56b6-45e1-80de-debe10e30cca\") " pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" Jul 06 21:13:33.327993 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.327966 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8d7g\" (UniqueName: \"kubernetes.io/projected/786193aa-77dd-42c4-8944-0aadc3a864f0-kube-api-access-r8d7g\") pod \"service-ca-operator-5d46d9ffd4-9qb5m\" (UID: \"786193aa-77dd-42c4-8944-0aadc3a864f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9qb5m" Jul 06 21:13:33.328091 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.328070 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlhlj\" (UniqueName: \"kubernetes.io/projected/68f8fa10-1463-4367-be99-23850541f8fc-kube-api-access-nlhlj\") pod \"volume-data-source-validator-5d456fdbcd-629h4\" (UID: \"68f8fa10-1463-4367-be99-23850541f8fc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-629h4" Jul 06 21:13:33.329652 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.329626 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29wvt\" (UniqueName: \"kubernetes.io/projected/2a156ee3-56b6-45e1-80de-debe10e30cca-kube-api-access-29wvt\") pod \"insights-operator-5c847dcfc4-cnsh7\" (UID: \"2a156ee3-56b6-45e1-80de-debe10e30cca\") " pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" Jul 06 21:13:33.418111 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.418077 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9qb5m" Jul 06 21:13:33.419357 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.419311 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-default-certificate\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:33.419531 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.419506 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-stats-auth\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:33.419594 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.419547 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad697c3c-f1a2-4209-827e-76d6164bd893-service-ca-bundle\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:33.419594 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.419583 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-metrics-certs\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:33.419699 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.419662 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgqxl\" (UniqueName: \"kubernetes.io/projected/ad697c3c-f1a2-4209-827e-76d6164bd893-kube-api-access-mgqxl\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:33.420053 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:33.419940 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad697c3c-f1a2-4209-827e-76d6164bd893-service-ca-bundle podName:ad697c3c-f1a2-4209-827e-76d6164bd893 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:33.919916471 +0000 UTC m=+38.682829465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ad697c3c-f1a2-4209-827e-76d6164bd893-service-ca-bundle") pod "router-default-548d5d67c5-5qdmw" (UID: "ad697c3c-f1a2-4209-827e-76d6164bd893") : configmap references non-existent config key: service-ca.crt Jul 06 21:13:33.420053 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:33.419963 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 21:13:33.420053 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:33.420024 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-metrics-certs podName:ad697c3c-f1a2-4209-827e-76d6164bd893 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:33.920006651 +0000 UTC m=+38.682919634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-metrics-certs") pod "router-default-548d5d67c5-5qdmw" (UID: "ad697c3c-f1a2-4209-827e-76d6164bd893") : secret "router-metrics-certs-default" not found Jul 06 21:13:33.423221 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.423202 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-default-certificate\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:33.425061 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.425020 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-stats-auth\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:33.429179 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.429149 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgqxl\" (UniqueName: \"kubernetes.io/projected/ad697c3c-f1a2-4209-827e-76d6164bd893-kube-api-access-mgqxl\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:33.429681 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.429661 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" Jul 06 21:13:33.444063 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.444041 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2tsvl"] Jul 06 21:13:33.448073 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:13:33.448050 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b6dfda_6ca5_4f85_a0f4_ef6a4e1b60be.slice/crio-1009f66369e5310dfb474b43c0937a1c0b4e4d5bcb844f00a7c18e2caa749464 WatchSource:0}: Error finding container 1009f66369e5310dfb474b43c0937a1c0b4e4d5bcb844f00a7c18e2caa749464: Status 404 returned error can't find the container with id 1009f66369e5310dfb474b43c0937a1c0b4e4d5bcb844f00a7c18e2caa749464 Jul 06 21:13:33.453577 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.453388 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-629h4" Jul 06 21:13:33.607600 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.607528 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-cnsh7"] Jul 06 21:13:33.612080 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:13:33.612047 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a156ee3_56b6_45e1_80de_debe10e30cca.slice/crio-225eccf0a6b4962e72c5a77aaa0dbf06097408ff64c4f041cde66ecab39bbe18 WatchSource:0}: Error finding container 225eccf0a6b4962e72c5a77aaa0dbf06097408ff64c4f041cde66ecab39bbe18: Status 404 returned error can't find the container with id 225eccf0a6b4962e72c5a77aaa0dbf06097408ff64c4f041cde66ecab39bbe18 Jul 06 21:13:33.623560 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.623491 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-2m2h6"] Jul 06 21:13:33.624129 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.624067 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9qb5m"] Jul 06 21:13:33.627046 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:13:33.627017 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b9e0da_5047_4951_8240_59dfad0b3efa.slice/crio-4fb946a1724051e2693f9dfb8d772b869d68c30e039ff785d57a5e6a0848274f WatchSource:0}: Error finding container 4fb946a1724051e2693f9dfb8d772b869d68c30e039ff785d57a5e6a0848274f: Status 404 returned error can't find the container with id 4fb946a1724051e2693f9dfb8d772b869d68c30e039ff785d57a5e6a0848274f Jul 06 21:13:33.627962 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:13:33.627935 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786193aa_77dd_42c4_8944_0aadc3a864f0.slice/crio-a16b291ced52f6101306ecda7219214ef9cc9e26c9945d76d2d0934a4165d310 WatchSource:0}: Error finding container a16b291ced52f6101306ecda7219214ef9cc9e26c9945d76d2d0934a4165d310: Status 404 returned error can't find the container with id a16b291ced52f6101306ecda7219214ef9cc9e26c9945d76d2d0934a4165d310 Jul 06 21:13:33.638797 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.638776 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-629h4"] Jul 06 21:13:33.641498 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:13:33.641473 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f8fa10_1463_4367_be99_23850541f8fc.slice/crio-14e5c25cc9bce3b17e33ccad3b087945ded54b58698f3585580ca912b76fb0ef WatchSource:0}: Error finding container 14e5c25cc9bce3b17e33ccad3b087945ded54b58698f3585580ca912b76fb0ef: Status 404 returned error can't find the container with id 14e5c25cc9bce3b17e33ccad3b087945ded54b58698f3585580ca912b76fb0ef Jul 06 21:13:33.722326 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.722290 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:33.722504 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:33.722463 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 21:13:33.722504 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:33.722485 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85bbcdb746-lph2c: secret "image-registry-tls" not found Jul 06 21:13:33.722625 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:33.722553 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls podName:da180245-4bba-4cbd-8bb6-6e370897c52d nodeName:}" failed. No retries permitted until 2026-07-06 21:13:34.722532267 +0000 UTC m=+39.485445274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls") pod "image-registry-85bbcdb746-lph2c" (UID: "da180245-4bba-4cbd-8bb6-6e370897c52d") : secret "image-registry-tls" not found Jul 06 21:13:33.924710 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.924618 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad697c3c-f1a2-4209-827e-76d6164bd893-service-ca-bundle\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:33.924710 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:33.924682 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-metrics-certs\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:33.924913 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:33.924804 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad697c3c-f1a2-4209-827e-76d6164bd893-service-ca-bundle podName:ad697c3c-f1a2-4209-827e-76d6164bd893 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:34.92477956 +0000 UTC m=+39.687692562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ad697c3c-f1a2-4209-827e-76d6164bd893-service-ca-bundle") pod "router-default-548d5d67c5-5qdmw" (UID: "ad697c3c-f1a2-4209-827e-76d6164bd893") : configmap references non-existent config key: service-ca.crt Jul 06 21:13:33.924913 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:33.924824 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 21:13:33.924913 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:33.924868 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-metrics-certs podName:ad697c3c-f1a2-4209-827e-76d6164bd893 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:34.924854641 +0000 UTC m=+39.687767640 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-metrics-certs") pod "router-default-548d5d67c5-5qdmw" (UID: "ad697c3c-f1a2-4209-827e-76d6164bd893") : secret "router-metrics-certs-default" not found Jul 06 21:13:34.054127 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:34.053896 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2m2h6" event={"ID":"a0b9e0da-5047-4951-8240-59dfad0b3efa","Type":"ContainerStarted","Data":"4fb946a1724051e2693f9dfb8d772b869d68c30e039ff785d57a5e6a0848274f"} Jul 06 21:13:34.055700 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:34.055672 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-629h4" event={"ID":"68f8fa10-1463-4367-be99-23850541f8fc","Type":"ContainerStarted","Data":"14e5c25cc9bce3b17e33ccad3b087945ded54b58698f3585580ca912b76fb0ef"} Jul 06 21:13:34.057067 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:34.057032 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9qb5m" event={"ID":"786193aa-77dd-42c4-8944-0aadc3a864f0","Type":"ContainerStarted","Data":"a16b291ced52f6101306ecda7219214ef9cc9e26c9945d76d2d0934a4165d310"} Jul 06 21:13:34.058413 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:34.058392 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2tsvl" event={"ID":"b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be","Type":"ContainerStarted","Data":"1009f66369e5310dfb474b43c0937a1c0b4e4d5bcb844f00a7c18e2caa749464"} Jul 06 21:13:34.059537 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:34.059512 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" event={"ID":"2a156ee3-56b6-45e1-80de-debe10e30cca","Type":"ContainerStarted","Data":"225eccf0a6b4962e72c5a77aaa0dbf06097408ff64c4f041cde66ecab39bbe18"} Jul 06 21:13:34.062478 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:34.062456 2581 generic.go:356] "Generic (PLEG): container finished" podID="b03516a3-66d9-4cd4-b814-dc23dd3bd9a1" containerID="86268635f42ebf5c3334ec0d6d6a79c9c7ef3ae68555dc42ae17aa0222d93076" exitCode=0 Jul 06 21:13:34.062552 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:34.062493 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x4grf" event={"ID":"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1","Type":"ContainerDied","Data":"86268635f42ebf5c3334ec0d6d6a79c9c7ef3ae68555dc42ae17aa0222d93076"} Jul 06 21:13:34.735645 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:34.734611 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:34.735645 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:34.734679 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c160ad01-8d36-4567-945d-fd6810e76537-metrics-tls\") pod \"dns-default-sstxl\" (UID: \"c160ad01-8d36-4567-945d-fd6810e76537\") " pod="openshift-dns/dns-default-sstxl" Jul 06 21:13:34.735645 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:34.734949 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 21:13:34.735645 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:34.735022 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c160ad01-8d36-4567-945d-fd6810e76537-metrics-tls podName:c160ad01-8d36-4567-945d-fd6810e76537 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:38.734999614 +0000 UTC m=+43.497912616 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c160ad01-8d36-4567-945d-fd6810e76537-metrics-tls") pod "dns-default-sstxl" (UID: "c160ad01-8d36-4567-945d-fd6810e76537") : secret "dns-default-metrics-tls" not found Jul 06 21:13:34.735645 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:34.735490 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 21:13:34.735645 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:34.735508 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85bbcdb746-lph2c: secret "image-registry-tls" not found Jul 06 21:13:34.735645 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:34.735559 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls podName:da180245-4bba-4cbd-8bb6-6e370897c52d nodeName:}" failed. No retries permitted until 2026-07-06 21:13:36.735543389 +0000 UTC m=+41.498456388 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls") pod "image-registry-85bbcdb746-lph2c" (UID: "da180245-4bba-4cbd-8bb6-6e370897c52d") : secret "image-registry-tls" not found Jul 06 21:13:34.938329 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:34.937450 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad697c3c-f1a2-4209-827e-76d6164bd893-service-ca-bundle\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:34.938329 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:34.937570 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-metrics-certs\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:34.938329 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:34.937790 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 21:13:34.938329 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:34.937856 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-metrics-certs podName:ad697c3c-f1a2-4209-827e-76d6164bd893 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:36.937836958 +0000 UTC m=+41.700749943 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-metrics-certs") pod "router-default-548d5d67c5-5qdmw" (UID: "ad697c3c-f1a2-4209-827e-76d6164bd893") : secret "router-metrics-certs-default" not found Jul 06 21:13:34.938694 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:34.938366 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad697c3c-f1a2-4209-827e-76d6164bd893-service-ca-bundle podName:ad697c3c-f1a2-4209-827e-76d6164bd893 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:36.938332951 +0000 UTC m=+41.701245954 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ad697c3c-f1a2-4209-827e-76d6164bd893-service-ca-bundle") pod "router-default-548d5d67c5-5qdmw" (UID: "ad697c3c-f1a2-4209-827e-76d6164bd893") : configmap references non-existent config key: service-ca.crt Jul 06 21:13:35.070327 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:35.070242 2581 generic.go:356] "Generic (PLEG): container finished" podID="b03516a3-66d9-4cd4-b814-dc23dd3bd9a1" containerID="2731be2411a82670baa4df4e298d731ec655bdce3717fe31a9966ee83311afdd" exitCode=0 Jul 06 21:13:35.070327 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:35.070307 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x4grf" event={"ID":"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1","Type":"ContainerDied","Data":"2731be2411a82670baa4df4e298d731ec655bdce3717fe31a9966ee83311afdd"} Jul 06 21:13:36.754048 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:36.754014 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:36.754428 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:36.754169 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 21:13:36.754428 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:36.754190 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85bbcdb746-lph2c: secret "image-registry-tls" not found Jul 06 21:13:36.754428 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:36.754245 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls podName:da180245-4bba-4cbd-8bb6-6e370897c52d nodeName:}" failed. No retries permitted until 2026-07-06 21:13:40.75423116 +0000 UTC m=+45.517144160 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls") pod "image-registry-85bbcdb746-lph2c" (UID: "da180245-4bba-4cbd-8bb6-6e370897c52d") : secret "image-registry-tls" not found Jul 06 21:13:36.956013 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:36.955975 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad697c3c-f1a2-4209-827e-76d6164bd893-service-ca-bundle\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:36.956013 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:36.956022 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-metrics-certs\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:36.956199 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:36.956124 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 21:13:36.956199 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:36.956146 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad697c3c-f1a2-4209-827e-76d6164bd893-service-ca-bundle podName:ad697c3c-f1a2-4209-827e-76d6164bd893 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:40.956129612 +0000 UTC m=+45.719042594 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ad697c3c-f1a2-4209-827e-76d6164bd893-service-ca-bundle") pod "router-default-548d5d67c5-5qdmw" (UID: "ad697c3c-f1a2-4209-827e-76d6164bd893") : configmap references non-existent config key: service-ca.crt Jul 06 21:13:36.956199 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:36.956170 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-metrics-certs podName:ad697c3c-f1a2-4209-827e-76d6164bd893 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:40.956163815 +0000 UTC m=+45.719076797 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-metrics-certs") pod "router-default-548d5d67c5-5qdmw" (UID: "ad697c3c-f1a2-4209-827e-76d6164bd893") : secret "router-metrics-certs-default" not found Jul 06 21:13:38.771310 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:38.771223 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c160ad01-8d36-4567-945d-fd6810e76537-metrics-tls\") pod \"dns-default-sstxl\" (UID: \"c160ad01-8d36-4567-945d-fd6810e76537\") " pod="openshift-dns/dns-default-sstxl" Jul 06 21:13:38.771741 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:38.771388 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 21:13:38.771741 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:38.771458 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c160ad01-8d36-4567-945d-fd6810e76537-metrics-tls podName:c160ad01-8d36-4567-945d-fd6810e76537 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:46.771441144 +0000 UTC m=+51.534354126 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c160ad01-8d36-4567-945d-fd6810e76537-metrics-tls") pod "dns-default-sstxl" (UID: "c160ad01-8d36-4567-945d-fd6810e76537") : secret "dns-default-metrics-tls" not found Jul 06 21:13:40.088841 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:40.088806 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x4grf" event={"ID":"b03516a3-66d9-4cd4-b814-dc23dd3bd9a1","Type":"ContainerStarted","Data":"186640a46cc8d91b24895d3d17d372b0d5df0669ea1f10cf21914fe02255a29a"} Jul 06 21:13:40.120065 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:40.119893 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x4grf" podStartSLOduration=18.892341341 podStartE2EDuration="45.119875698s" totalImagesPullingTime="26.227534357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:12:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:40.118053778 +0000 UTC m=+44.880966820" watchObservedRunningTime="2026-07-06 21:13:40.119875698 +0000 UTC m=+44.882788704" Jul 06 21:13:40.790865 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:40.790819 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:40.791132 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:40.790985 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 21:13:40.791132 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:40.791010 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85bbcdb746-lph2c: secret "image-registry-tls" not found Jul 06 21:13:40.791132 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:40.791082 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls podName:da180245-4bba-4cbd-8bb6-6e370897c52d nodeName:}" failed. No retries permitted until 2026-07-06 21:13:48.791060667 +0000 UTC m=+53.553973668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls") pod "image-registry-85bbcdb746-lph2c" (UID: "da180245-4bba-4cbd-8bb6-6e370897c52d") : secret "image-registry-tls" not found Jul 06 21:13:40.992413 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:40.992374 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad697c3c-f1a2-4209-827e-76d6164bd893-service-ca-bundle\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:40.992413 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:40.992420 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-metrics-certs\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:40.992725 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:40.992520 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 21:13:40.992725 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:40.992579 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad697c3c-f1a2-4209-827e-76d6164bd893-service-ca-bundle podName:ad697c3c-f1a2-4209-827e-76d6164bd893 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:48.992556486 +0000 UTC m=+53.755469482 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ad697c3c-f1a2-4209-827e-76d6164bd893-service-ca-bundle") pod "router-default-548d5d67c5-5qdmw" (UID: "ad697c3c-f1a2-4209-827e-76d6164bd893") : configmap references non-existent config key: service-ca.crt Jul 06 21:13:40.992725 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:40.992614 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-metrics-certs podName:ad697c3c-f1a2-4209-827e-76d6164bd893 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:48.992603034 +0000 UTC m=+53.755516021 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-metrics-certs") pod "router-default-548d5d67c5-5qdmw" (UID: "ad697c3c-f1a2-4209-827e-76d6164bd893") : secret "router-metrics-certs-default" not found Jul 06 21:13:41.095177 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.095089 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-629h4" event={"ID":"68f8fa10-1463-4367-be99-23850541f8fc","Type":"ContainerStarted","Data":"a52ef3f1bd93fcd8fd4ab738120bfae642875ec56390b6fe8c992aa08be379cd"} Jul 06 21:13:41.097227 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.097201 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9qb5m" event={"ID":"786193aa-77dd-42c4-8944-0aadc3a864f0","Type":"ContainerStarted","Data":"0b41b6888222a3231d3006394b5371be70ea8a576292a5a950b9bee1bee6b54e"} Jul 06 21:13:41.098883 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.098857 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jxz5h" event={"ID":"704cb9c1-1b35-419a-90f0-0cf0dd5498f5","Type":"ContainerStarted","Data":"bd1fc244a9ba36287d038e5bc93ec4d060d0a3dcf24f33b84fd49ba7a3ea7192"} Jul 06 21:13:41.100493 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.100472 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2tsvl" event={"ID":"b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be","Type":"ContainerStarted","Data":"c96e15ae2857b8aee88aa31a76397a63a7c3784cf723b00c55870a7492e00b12"} Jul 06 21:13:41.102111 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.102060 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" event={"ID":"2a156ee3-56b6-45e1-80de-debe10e30cca","Type":"ContainerStarted","Data":"bf571484cc00c9454462354799ce0637463ddccbbadb69e956202230a43ab5e7"} Jul 06 21:13:41.103654 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.103631 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2m2h6" event={"ID":"a0b9e0da-5047-4951-8240-59dfad0b3efa","Type":"ContainerStarted","Data":"b25fd2f97daa2f81e1964a974325ba90bdde9fa2ce9c348d33bf5607511fa62c"} Jul 06 21:13:41.110241 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.110199 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-629h4" podStartSLOduration=1.801220992 podStartE2EDuration="8.110183908s" totalImagesPullingTime="6.308962916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:41.110075985 +0000 UTC m=+45.872988988" watchObservedRunningTime="2026-07-06 21:13:41.110183908 +0000 UTC m=+45.873096914" Jul 06 21:13:41.128158 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.128115 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2tsvl" podStartSLOduration=2.625644589 podStartE2EDuration="9.128103059s" totalImagesPullingTime="6.50245847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:13:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:41.127336155 +0000 UTC m=+45.890249160" watchObservedRunningTime="2026-07-06 21:13:41.128103059 +0000 UTC m=+45.891016063" Jul 06 21:13:41.147738 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.147692 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9qb5m" podStartSLOduration=1.8257006310000001 podStartE2EDuration="8.147675742s" totalImagesPullingTime="6.321975111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:41.147209933 +0000 UTC m=+45.910122938" watchObservedRunningTime="2026-07-06 21:13:41.147675742 +0000 UTC m=+45.910588747" Jul 06 21:13:41.171304 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.171256 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" podStartSLOduration=1.8324789849999998 podStartE2EDuration="8.171238099s" totalImagesPullingTime="6.338759114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:41.170656853 +0000 UTC m=+45.933569859" watchObservedRunningTime="2026-07-06 21:13:41.171238099 +0000 UTC m=+45.934151104" Jul 06 21:13:41.188125 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.187703 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2m2h6" podStartSLOduration=2.852391518 podStartE2EDuration="9.187688897s" totalImagesPullingTime="6.335297379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:13:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:41.186707808 +0000 UTC m=+45.949620813" watchObservedRunningTime="2026-07-06 21:13:41.187688897 +0000 UTC m=+45.950601901" Jul 06 21:13:41.210396 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.210319 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jxz5h" podStartSLOduration=2.797778729 podStartE2EDuration="11.210299283s" totalImagesPullingTime="8.412520554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:13:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:13:41.209598482 +0000 UTC m=+45.972511484" watchObservedRunningTime="2026-07-06 21:13:41.210299283 +0000 UTC m=+45.973212289" Jul 06 21:13:41.358988 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.358951 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b9wqf"] Jul 06 21:13:41.392098 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.392060 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b9wqf"] Jul 06 21:13:41.392247 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.392178 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:41.395041 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.395009 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 21:13:41.395422 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.395286 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 21:13:41.395422 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.395324 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-spnsv\"" Jul 06 21:13:41.497078 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.497045 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:41.497078 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.497083 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-crio-socket\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:41.497329 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.497157 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psv2h\" (UniqueName: \"kubernetes.io/projected/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-kube-api-access-psv2h\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:41.497329 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.497205 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-data-volume\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:41.497329 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.497289 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:41.597865 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.597824 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:41.598047 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.597885 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-crio-socket\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:41.598047 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.597942 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psv2h\" (UniqueName: \"kubernetes.io/projected/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-kube-api-access-psv2h\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:41.598047 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:41.597950 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 21:13:41.598047 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.597981 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-data-volume\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:41.598047 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:41.598026 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-kube-rbac-proxy-cm podName:4c41ac7d-dd13-4ea9-beec-7dfa17111ba1 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:42.098006322 +0000 UTC m=+46.860919322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b9wqf" (UID: "4c41ac7d-dd13-4ea9-beec-7dfa17111ba1") : configmap "kube-rbac-proxy" not found Jul 06 21:13:41.598319 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.598133 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:41.598319 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.598146 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-crio-socket\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:41.598319 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:41.598242 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 21:13:41.598319 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:41.598276 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-insights-runtime-extractor-tls podName:4c41ac7d-dd13-4ea9-beec-7dfa17111ba1 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:42.098264725 +0000 UTC m=+46.861177707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b9wqf" (UID: "4c41ac7d-dd13-4ea9-beec-7dfa17111ba1") : secret "insights-runtime-extractor-tls" not found Jul 06 21:13:41.598512 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.598324 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-data-volume\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:41.607463 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:41.607432 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psv2h\" (UniqueName: \"kubernetes.io/projected/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-kube-api-access-psv2h\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:42.102200 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:42.102162 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:42.102607 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:42.102214 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:42.102607 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:42.102300 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 21:13:42.102607 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:42.102312 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 21:13:42.102607 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:42.102374 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-kube-rbac-proxy-cm podName:4c41ac7d-dd13-4ea9-beec-7dfa17111ba1 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:43.102358777 +0000 UTC m=+47.865271774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b9wqf" (UID: "4c41ac7d-dd13-4ea9-beec-7dfa17111ba1") : configmap "kube-rbac-proxy" not found Jul 06 21:13:42.102607 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:42.102388 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-insights-runtime-extractor-tls podName:4c41ac7d-dd13-4ea9-beec-7dfa17111ba1 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:43.102382159 +0000 UTC m=+47.865295141 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b9wqf" (UID: "4c41ac7d-dd13-4ea9-beec-7dfa17111ba1") : secret "insights-runtime-extractor-tls" not found Jul 06 21:13:42.781136 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:42.781103 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2q95f_febe62f7-bf85-42c1-8902-c4dabb9ba553/dns-node-resolver/0.log" Jul 06 21:13:43.111702 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:43.111672 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:43.112081 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:43.111789 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 21:13:43.112081 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:43.111859 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-kube-rbac-proxy-cm podName:4c41ac7d-dd13-4ea9-beec-7dfa17111ba1 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:45.111843986 +0000 UTC m=+49.874756972 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b9wqf" (UID: "4c41ac7d-dd13-4ea9-beec-7dfa17111ba1") : configmap "kube-rbac-proxy" not found Jul 06 21:13:43.112081 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:43.111894 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:43.112081 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:43.112001 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 21:13:43.112081 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:43.112032 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-insights-runtime-extractor-tls podName:4c41ac7d-dd13-4ea9-beec-7dfa17111ba1 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:45.112024468 +0000 UTC m=+49.874937449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b9wqf" (UID: "4c41ac7d-dd13-4ea9-beec-7dfa17111ba1") : secret "insights-runtime-extractor-tls" not found Jul 06 21:13:44.181291 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:44.181263 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lds24_08f54334-35ee-460d-ae74-75618e3e13c5/node-ca/0.log" Jul 06 21:13:44.784115 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:44.784087 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jxz5h_704cb9c1-1b35-419a-90f0-0cf0dd5498f5/serve-healthcheck-canary/0.log" Jul 06 21:13:45.131980 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:45.131949 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:45.132129 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:45.132069 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 21:13:45.132195 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:45.132143 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-kube-rbac-proxy-cm podName:4c41ac7d-dd13-4ea9-beec-7dfa17111ba1 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:49.132125908 +0000 UTC m=+53.895038908 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b9wqf" (UID: "4c41ac7d-dd13-4ea9-beec-7dfa17111ba1") : configmap "kube-rbac-proxy" not found Jul 06 21:13:45.132237 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:45.132203 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:45.132361 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:45.132320 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 21:13:45.132428 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:45.132416 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-insights-runtime-extractor-tls podName:4c41ac7d-dd13-4ea9-beec-7dfa17111ba1 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:49.132399712 +0000 UTC m=+53.895312698 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b9wqf" (UID: "4c41ac7d-dd13-4ea9-beec-7dfa17111ba1") : secret "insights-runtime-extractor-tls" not found Jul 06 21:13:46.846312 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:46.846276 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c160ad01-8d36-4567-945d-fd6810e76537-metrics-tls\") pod \"dns-default-sstxl\" (UID: \"c160ad01-8d36-4567-945d-fd6810e76537\") " pod="openshift-dns/dns-default-sstxl" Jul 06 21:13:46.846714 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:46.846440 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 21:13:46.846714 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:46.846504 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c160ad01-8d36-4567-945d-fd6810e76537-metrics-tls podName:c160ad01-8d36-4567-945d-fd6810e76537 nodeName:}" failed. No retries permitted until 2026-07-06 21:14:02.846488975 +0000 UTC m=+67.609401957 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c160ad01-8d36-4567-945d-fd6810e76537-metrics-tls") pod "dns-default-sstxl" (UID: "c160ad01-8d36-4567-945d-fd6810e76537") : secret "dns-default-metrics-tls" not found Jul 06 21:13:48.863825 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:48.863785 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:13:48.864221 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:48.863940 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 21:13:48.864221 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:48.863960 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85bbcdb746-lph2c: secret "image-registry-tls" not found Jul 06 21:13:48.864221 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:48.864026 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls podName:da180245-4bba-4cbd-8bb6-6e370897c52d nodeName:}" failed. No retries permitted until 2026-07-06 21:14:04.864009818 +0000 UTC m=+69.626922800 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls") pod "image-registry-85bbcdb746-lph2c" (UID: "da180245-4bba-4cbd-8bb6-6e370897c52d") : secret "image-registry-tls" not found Jul 06 21:13:49.065649 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:49.065615 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad697c3c-f1a2-4209-827e-76d6164bd893-service-ca-bundle\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:49.065807 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:49.065660 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-metrics-certs\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:13:49.065807 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:49.065767 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 21:13:49.065893 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:49.065816 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-metrics-certs podName:ad697c3c-f1a2-4209-827e-76d6164bd893 nodeName:}" failed. No retries permitted until 2026-07-06 21:14:05.065803709 +0000 UTC m=+69.828716695 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-metrics-certs") pod "router-default-548d5d67c5-5qdmw" (UID: "ad697c3c-f1a2-4209-827e-76d6164bd893") : secret "router-metrics-certs-default" not found Jul 06 21:13:49.065893 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:49.065834 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad697c3c-f1a2-4209-827e-76d6164bd893-service-ca-bundle podName:ad697c3c-f1a2-4209-827e-76d6164bd893 nodeName:}" failed. No retries permitted until 2026-07-06 21:14:05.06582781 +0000 UTC m=+69.828740792 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ad697c3c-f1a2-4209-827e-76d6164bd893-service-ca-bundle") pod "router-default-548d5d67c5-5qdmw" (UID: "ad697c3c-f1a2-4209-827e-76d6164bd893") : configmap references non-existent config key: service-ca.crt Jul 06 21:13:49.166692 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:49.166592 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:49.166692 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:49.166656 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:49.166901 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:49.166738 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 21:13:49.166901 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:13:49.166818 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-insights-runtime-extractor-tls podName:4c41ac7d-dd13-4ea9-beec-7dfa17111ba1 nodeName:}" failed. No retries permitted until 2026-07-06 21:13:57.166804102 +0000 UTC m=+61.929717084 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b9wqf" (UID: "4c41ac7d-dd13-4ea9-beec-7dfa17111ba1") : secret "insights-runtime-extractor-tls" not found Jul 06 21:13:49.167239 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:49.167220 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:57.046066 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:57.046039 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq68j" Jul 06 21:13:57.233685 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:57.233642 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:57.235874 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:57.235852 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c41ac7d-dd13-4ea9-beec-7dfa17111ba1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9wqf\" (UID: \"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1\") " pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:57.304607 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:57.304520 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-spnsv\"" Jul 06 21:13:57.312715 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:57.312686 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b9wqf" Jul 06 21:13:57.437604 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:57.437574 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b9wqf"] Jul 06 21:13:57.440386 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:13:57.440360 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c41ac7d_dd13_4ea9_beec_7dfa17111ba1.slice/crio-e8d5cc012f0893fbe755db73ed166164f5e19774fa160aee68ab21343c981ede WatchSource:0}: Error finding container e8d5cc012f0893fbe755db73ed166164f5e19774fa160aee68ab21343c981ede: Status 404 returned error can't find the container with id e8d5cc012f0893fbe755db73ed166164f5e19774fa160aee68ab21343c981ede Jul 06 21:13:58.159870 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:13:58.159805 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9wqf" event={"ID":"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1","Type":"ContainerStarted","Data":"e8d5cc012f0893fbe755db73ed166164f5e19774fa160aee68ab21343c981ede"} Jul 06 21:14:01.170673 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:01.170640 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9wqf" event={"ID":"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1","Type":"ContainerStarted","Data":"cf9ad633802c307d7d74d2bf623b506d75904fdeda0886fb3602af1006a7f680"} Jul 06 21:14:01.471472 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:01.471434 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw798\" (UniqueName: \"kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798\") pod \"network-check-target-wjc92\" (UID: \"b4d5a450-11db-44a5-bb22-35f668de29c7\") " pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:14:01.471641 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:01.471500 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs\") pod \"network-metrics-daemon-79rs9\" (UID: \"e4f4e605-6394-4afd-8ef5-55c6fe78cb07\") " pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:14:01.473866 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:01.473842 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 21:14:01.474028 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:01.474006 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw798\" (UniqueName: \"kubernetes.io/projected/b4d5a450-11db-44a5-bb22-35f668de29c7-kube-api-access-pw798\") pod \"network-check-target-wjc92\" (UID: \"b4d5a450-11db-44a5-bb22-35f668de29c7\") " pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:14:01.483734 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:01.483713 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f4e605-6394-4afd-8ef5-55c6fe78cb07-metrics-certs\") pod \"network-metrics-daemon-79rs9\" (UID: \"e4f4e605-6394-4afd-8ef5-55c6fe78cb07\") " pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:14:01.495935 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:01.495914 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kz87d\"" Jul 06 21:14:01.503948 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:01.503928 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:14:01.507623 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:01.507609 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c2dbc\"" Jul 06 21:14:01.516465 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:01.516445 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79rs9" Jul 06 21:14:01.575463 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:01.572522 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret\") pod \"global-pull-secret-syncer-zn2lg\" (UID: \"e4aacd25-63b3-4414-8e53-5d418169367b\") " pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:14:01.575559 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:01.575522 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 21:14:01.587134 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:01.587076 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4aacd25-63b3-4414-8e53-5d418169367b-original-pull-secret\") pod \"global-pull-secret-syncer-zn2lg\" (UID: \"e4aacd25-63b3-4414-8e53-5d418169367b\") " pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:14:01.644125 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:01.644093 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wjc92"] Jul 06 21:14:01.662714 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:01.662688 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-79rs9"] Jul 06 21:14:01.709118 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:14:01.709080 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d5a450_11db_44a5_bb22_35f668de29c7.slice/crio-1f56901af0f248d71caf34ed8ff96eb4a2dbe1ffbf0618757750b776f5e44182 WatchSource:0}: Error finding container 1f56901af0f248d71caf34ed8ff96eb4a2dbe1ffbf0618757750b776f5e44182: Status 404 returned error can't find the container with id 1f56901af0f248d71caf34ed8ff96eb4a2dbe1ffbf0618757750b776f5e44182 Jul 06 21:14:01.709789 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:14:01.709724 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f4e605_6394_4afd_8ef5_55c6fe78cb07.slice/crio-c814639a145159ea891b105b416f382cdb8b4032bdd625d0a801bf5eb56a5c7d WatchSource:0}: Error finding container c814639a145159ea891b105b416f382cdb8b4032bdd625d0a801bf5eb56a5c7d: Status 404 returned error can't find the container with id c814639a145159ea891b105b416f382cdb8b4032bdd625d0a801bf5eb56a5c7d Jul 06 21:14:01.800264 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:01.800227 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zn2lg" Jul 06 21:14:01.919645 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:01.919615 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zn2lg"] Jul 06 21:14:01.922167 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:14:01.922137 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4aacd25_63b3_4414_8e53_5d418169367b.slice/crio-78cb9c3fd9675e189736cf25b7fc0aba9c4b1cca733cb0dd7749f73fb3f1ecc2 WatchSource:0}: Error finding container 78cb9c3fd9675e189736cf25b7fc0aba9c4b1cca733cb0dd7749f73fb3f1ecc2: Status 404 returned error can't find the container with id 78cb9c3fd9675e189736cf25b7fc0aba9c4b1cca733cb0dd7749f73fb3f1ecc2 Jul 06 21:14:02.176782 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:02.176747 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9wqf" event={"ID":"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1","Type":"ContainerStarted","Data":"f098351faddaa27f25f4f4cb637327a26945a00760f9da6647628be8cbd2efad"} Jul 06 21:14:02.178102 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:02.178071 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zn2lg" event={"ID":"e4aacd25-63b3-4414-8e53-5d418169367b","Type":"ContainerStarted","Data":"78cb9c3fd9675e189736cf25b7fc0aba9c4b1cca733cb0dd7749f73fb3f1ecc2"} Jul 06 21:14:02.179462 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:02.179437 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-79rs9" event={"ID":"e4f4e605-6394-4afd-8ef5-55c6fe78cb07","Type":"ContainerStarted","Data":"c814639a145159ea891b105b416f382cdb8b4032bdd625d0a801bf5eb56a5c7d"} Jul 06 21:14:02.181051 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:02.181029 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wjc92" event={"ID":"b4d5a450-11db-44a5-bb22-35f668de29c7","Type":"ContainerStarted","Data":"fbd0e1c81b2c7c0b18852dd0fe58d4d1f2cc39048f64f547e3cdc228a6793c6b"} Jul 06 21:14:02.181154 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:02.181071 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wjc92" event={"ID":"b4d5a450-11db-44a5-bb22-35f668de29c7","Type":"ContainerStarted","Data":"1f56901af0f248d71caf34ed8ff96eb4a2dbe1ffbf0618757750b776f5e44182"} Jul 06 21:14:02.181228 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:02.181199 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:14:02.197053 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:02.197004 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wjc92" podStartSLOduration=66.196984332 podStartE2EDuration="1m6.196984332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:14:02.196018718 +0000 UTC m=+66.958931723" watchObservedRunningTime="2026-07-06 21:14:02.196984332 +0000 UTC m=+66.959897335" Jul 06 21:14:02.882940 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:02.882891 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c160ad01-8d36-4567-945d-fd6810e76537-metrics-tls\") pod \"dns-default-sstxl\" (UID: \"c160ad01-8d36-4567-945d-fd6810e76537\") " pod="openshift-dns/dns-default-sstxl" Jul 06 21:14:02.886769 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:02.886738 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c160ad01-8d36-4567-945d-fd6810e76537-metrics-tls\") pod \"dns-default-sstxl\" (UID: \"c160ad01-8d36-4567-945d-fd6810e76537\") " pod="openshift-dns/dns-default-sstxl" Jul 06 21:14:03.141389 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:03.141302 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4w8fd\"" Jul 06 21:14:03.149443 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:03.149409 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sstxl" Jul 06 21:14:04.768457 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:04.768377 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sstxl"] Jul 06 21:14:04.771563 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:14:04.771532 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc160ad01_8d36_4567_945d_fd6810e76537.slice/crio-6e40589167c58e4d92fa3b93ce4bbadf0b1ad21e026111bbfe783734681f5546 WatchSource:0}: Error finding container 6e40589167c58e4d92fa3b93ce4bbadf0b1ad21e026111bbfe783734681f5546: Status 404 returned error can't find the container with id 6e40589167c58e4d92fa3b93ce4bbadf0b1ad21e026111bbfe783734681f5546 Jul 06 21:14:04.898453 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:04.898409 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:14:04.901079 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:04.901053 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls\") pod \"image-registry-85bbcdb746-lph2c\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:14:05.100241 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:05.100209 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad697c3c-f1a2-4209-827e-76d6164bd893-service-ca-bundle\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:14:05.100440 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:05.100256 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-metrics-certs\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:14:05.101036 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:05.101007 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad697c3c-f1a2-4209-827e-76d6164bd893-service-ca-bundle\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:14:05.102935 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:05.102913 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad697c3c-f1a2-4209-827e-76d6164bd893-metrics-certs\") pod \"router-default-548d5d67c5-5qdmw\" (UID: \"ad697c3c-f1a2-4209-827e-76d6164bd893\") " pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:14:05.112150 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:05.112127 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dgf9v\"" Jul 06 21:14:05.120271 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:05.120241 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:14:05.196295 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:05.196258 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9wqf" event={"ID":"4c41ac7d-dd13-4ea9-beec-7dfa17111ba1","Type":"ContainerStarted","Data":"4c008f0083f367ca927ba9989275ec1cee86da970c28c8a3c4dd05c7fedaad68"} Jul 06 21:14:05.197625 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:05.197590 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sstxl" event={"ID":"c160ad01-8d36-4567-945d-fd6810e76537","Type":"ContainerStarted","Data":"6e40589167c58e4d92fa3b93ce4bbadf0b1ad21e026111bbfe783734681f5546"} Jul 06 21:14:05.199452 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:05.199421 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-79rs9" event={"ID":"e4f4e605-6394-4afd-8ef5-55c6fe78cb07","Type":"ContainerStarted","Data":"2d66184c418170386b1e70a3f8b8da258473a9532e8b4a9172579ce8d051ce61"} Jul 06 21:14:05.199564 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:05.199469 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-79rs9" event={"ID":"e4f4e605-6394-4afd-8ef5-55c6fe78cb07","Type":"ContainerStarted","Data":"01253c655e812bdb3c629a0da184960432b23f3467dfc2333567eb3595c817d1"} Jul 06 21:14:05.217997 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:05.217939 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-b9wqf" podStartSLOduration=17.11601841 podStartE2EDuration="24.217921541s" totalImagesPullingTime="7.101903131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:13:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:14:05.217058601 +0000 UTC m=+69.979971609" watchObservedRunningTime="2026-07-06 21:14:05.217921541 +0000 UTC m=+69.980834547" Jul 06 21:14:05.233086 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:05.233037 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-79rs9" podStartSLOduration=67.353035554 podStartE2EDuration="1m10.233021918s" totalImagesPullingTime="2.879986364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:12:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:14:05.232452374 +0000 UTC m=+69.995365380" watchObservedRunningTime="2026-07-06 21:14:05.233021918 +0000 UTC m=+69.995934923" Jul 06 21:14:05.290430 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:05.290396 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9kv9l\"" Jul 06 21:14:05.298397 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:05.298361 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:14:06.314875 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:06.314840 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-548d5d67c5-5qdmw"] Jul 06 21:14:06.318650 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:14:06.318620 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad697c3c_f1a2_4209_827e_76d6164bd893.slice/crio-b6f5e7f59e289df5af01e51047e022ecad0e926270521f574c6d715a3fdea053 WatchSource:0}: Error finding container b6f5e7f59e289df5af01e51047e022ecad0e926270521f574c6d715a3fdea053: Status 404 returned error can't find the container with id b6f5e7f59e289df5af01e51047e022ecad0e926270521f574c6d715a3fdea053 Jul 06 21:14:06.331800 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:06.331774 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85bbcdb746-lph2c"] Jul 06 21:14:07.211213 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:07.211175 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zn2lg" event={"ID":"e4aacd25-63b3-4414-8e53-5d418169367b","Type":"ContainerStarted","Data":"3e8e9cfb2cb1e2c011f62fb16cf7952b8d7a9a2c36c31e042328da79edb5a45e"} Jul 06 21:14:07.213118 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:07.213082 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-548d5d67c5-5qdmw" event={"ID":"ad697c3c-f1a2-4209-827e-76d6164bd893","Type":"ContainerStarted","Data":"45a18f3f0e2f2e563cf98ce93c9def7a8e0f0dba7baeea733acd6ebf077a9a28"} Jul 06 21:14:07.213259 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:07.213134 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-548d5d67c5-5qdmw" event={"ID":"ad697c3c-f1a2-4209-827e-76d6164bd893","Type":"ContainerStarted","Data":"b6f5e7f59e289df5af01e51047e022ecad0e926270521f574c6d715a3fdea053"} Jul 06 21:14:07.214748 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:07.214720 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" event={"ID":"da180245-4bba-4cbd-8bb6-6e370897c52d","Type":"ContainerStarted","Data":"5d142067fc2755b68870cef59bb14055d42b065df54b3c011db5e5f01a5e7b72"} Jul 06 21:14:07.214871 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:07.214763 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" event={"ID":"da180245-4bba-4cbd-8bb6-6e370897c52d","Type":"ContainerStarted","Data":"8e2aeeceda87f74c51ac8699fbf49a1e4e5d57cfe872a6a0f7d1dfd41da04040"} Jul 06 21:14:07.214931 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:07.214877 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:14:07.229183 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:07.229130 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zn2lg" podStartSLOduration=66.945832148 podStartE2EDuration="1m11.229116707s" totalImagesPullingTime="4.283284559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:14:07.229062021 +0000 UTC m=+71.991975025" watchObservedRunningTime="2026-07-06 21:14:07.229116707 +0000 UTC m=+71.992029710" Jul 06 21:14:07.251318 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:07.251252 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-548d5d67c5-5qdmw" podStartSLOduration=34.251230159 podStartE2EDuration="34.251230159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:14:07.25086066 +0000 UTC m=+72.013773665" watchObservedRunningTime="2026-07-06 21:14:07.251230159 +0000 UTC m=+72.014143167" Jul 06 21:14:07.274140 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:07.274082 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" podStartSLOduration=35.274061794 podStartE2EDuration="35.274061794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:14:07.272677014 +0000 UTC m=+72.035590032" watchObservedRunningTime="2026-07-06 21:14:07.274061794 +0000 UTC m=+72.036974800" Jul 06 21:14:07.299192 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:07.299156 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:14:07.302310 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:07.302284 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:14:08.220458 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:08.220415 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sstxl" event={"ID":"c160ad01-8d36-4567-945d-fd6810e76537","Type":"ContainerStarted","Data":"40afcb34de3b9a26c259c4539fe49576f49fddbe67c03da96f266433d9a524a6"} Jul 06 21:14:08.220896 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:08.220476 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sstxl" event={"ID":"c160ad01-8d36-4567-945d-fd6810e76537","Type":"ContainerStarted","Data":"b19a9e870532155b8e47f633899402e0efefb6a0943bc451ed6ab4ab1815c9a1"} Jul 06 21:14:08.220974 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:08.220894 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sstxl" Jul 06 21:14:08.220974 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:08.220936 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:14:08.222047 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:08.222025 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-548d5d67c5-5qdmw" Jul 06 21:14:08.238170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:08.238129 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sstxl" podStartSLOduration=35.622092355 podStartE2EDuration="38.238116366s" totalImagesPullingTime="2.616024011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:13:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:14:08.236711944 +0000 UTC m=+72.999624949" watchObservedRunningTime="2026-07-06 21:14:08.238116366 +0000 UTC m=+73.001029370" Jul 06 21:14:08.449592 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:08.449563 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85bbcdb746-lph2c"] Jul 06 21:14:16.037576 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.037542 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74c76db575-thdzm"] Jul 06 21:14:16.040608 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.040591 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.047632 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.047612 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 21:14:16.048302 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.048288 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 21:14:16.048620 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.048596 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 21:14:16.048706 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.048656 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-d8hrj\"" Jul 06 21:14:16.048706 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.048696 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 21:14:16.048811 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.048701 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 21:14:16.048811 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.048732 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 21:14:16.048811 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.048697 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 21:14:16.056059 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.056023 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 21:14:16.062099 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.062079 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74c76db575-thdzm"] Jul 06 21:14:16.078785 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.078764 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7aec20eb-fd13-4135-95d0-ddea26dd1001-console-oauth-config\") pod \"console-74c76db575-thdzm\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.078876 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.078810 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7aec20eb-fd13-4135-95d0-ddea26dd1001-console-serving-cert\") pod \"console-74c76db575-thdzm\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.078876 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.078832 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-trusted-ca-bundle\") pod \"console-74c76db575-thdzm\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.078876 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.078851 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-service-ca\") pod \"console-74c76db575-thdzm\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.078975 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.078887 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94pvn\" (UniqueName: \"kubernetes.io/projected/7aec20eb-fd13-4135-95d0-ddea26dd1001-kube-api-access-94pvn\") pod \"console-74c76db575-thdzm\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.078975 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.078919 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-console-config\") pod \"console-74c76db575-thdzm\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.078975 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.078940 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-oauth-serving-cert\") pod \"console-74c76db575-thdzm\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.179972 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.179938 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7aec20eb-fd13-4135-95d0-ddea26dd1001-console-serving-cert\") pod \"console-74c76db575-thdzm\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.179972 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.179975 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-trusted-ca-bundle\") pod \"console-74c76db575-thdzm\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.180181 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.179992 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-service-ca\") pod \"console-74c76db575-thdzm\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.180181 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.180009 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94pvn\" (UniqueName: \"kubernetes.io/projected/7aec20eb-fd13-4135-95d0-ddea26dd1001-kube-api-access-94pvn\") pod \"console-74c76db575-thdzm\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.180181 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.180026 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-console-config\") pod \"console-74c76db575-thdzm\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.180181 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.180058 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-oauth-serving-cert\") pod \"console-74c76db575-thdzm\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.180181 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.180093 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7aec20eb-fd13-4135-95d0-ddea26dd1001-console-oauth-config\") pod \"console-74c76db575-thdzm\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.180827 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.180795 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-console-config\") pod \"console-74c76db575-thdzm\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.180827 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.180816 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-service-ca\") pod \"console-74c76db575-thdzm\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.181005 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.180872 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-oauth-serving-cert\") pod \"console-74c76db575-thdzm\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.181005 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.180933 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-trusted-ca-bundle\") pod \"console-74c76db575-thdzm\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.182504 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.182483 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7aec20eb-fd13-4135-95d0-ddea26dd1001-console-oauth-config\") pod \"console-74c76db575-thdzm\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.182616 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.182599 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7aec20eb-fd13-4135-95d0-ddea26dd1001-console-serving-cert\") pod \"console-74c76db575-thdzm\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.187984 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.187966 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94pvn\" (UniqueName: \"kubernetes.io/projected/7aec20eb-fd13-4135-95d0-ddea26dd1001-kube-api-access-94pvn\") pod \"console-74c76db575-thdzm\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.349295 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.349267 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:16.473318 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:16.473288 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74c76db575-thdzm"] Jul 06 21:14:16.476358 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:14:16.476322 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aec20eb_fd13_4135_95d0_ddea26dd1001.slice/crio-be06e14847c9629954135c33001efdaaaa128ecdf2d50f963a238195f5ad3d21 WatchSource:0}: Error finding container be06e14847c9629954135c33001efdaaaa128ecdf2d50f963a238195f5ad3d21: Status 404 returned error can't find the container with id be06e14847c9629954135c33001efdaaaa128ecdf2d50f963a238195f5ad3d21 Jul 06 21:14:17.252302 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:17.252265 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c76db575-thdzm" event={"ID":"7aec20eb-fd13-4135-95d0-ddea26dd1001","Type":"ContainerStarted","Data":"be06e14847c9629954135c33001efdaaaa128ecdf2d50f963a238195f5ad3d21"} Jul 06 21:14:19.228464 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:19.228319 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sstxl" Jul 06 21:14:20.263793 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:20.263755 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c76db575-thdzm" event={"ID":"7aec20eb-fd13-4135-95d0-ddea26dd1001","Type":"ContainerStarted","Data":"146ac90790f5c7f277f3e4001d794cd489a1b4a26d32104c5dec4a8240cee50e"} Jul 06 21:14:20.282484 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:20.282417 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74c76db575-thdzm" podStartSLOduration=1.527104123 podStartE2EDuration="4.282400551s" totalImagesPullingTime="2.755296428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:14:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:14:20.281602913 +0000 UTC m=+85.044515918" watchObservedRunningTime="2026-07-06 21:14:20.282400551 +0000 UTC m=+85.045313554" Jul 06 21:14:23.052933 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.052903 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ssmlt"] Jul 06 21:14:23.055245 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.055226 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.058200 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.058180 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 21:14:23.058316 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.058226 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 21:14:23.058316 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.058227 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 21:14:23.058426 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.058371 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 21:14:23.059078 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.059062 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 21:14:23.059156 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.059136 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6nbt4\"" Jul 06 21:14:23.059156 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.059150 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 21:14:23.147524 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.147488 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2b7070b6-7594-44c9-9d25-6ef3366255bf-root\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.147693 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.147535 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfc7f\" (UniqueName: \"kubernetes.io/projected/2b7070b6-7594-44c9-9d25-6ef3366255bf-kube-api-access-nfc7f\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.147693 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.147564 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2b7070b6-7594-44c9-9d25-6ef3366255bf-node-exporter-accelerators-collector-config\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.147693 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.147580 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2b7070b6-7594-44c9-9d25-6ef3366255bf-node-exporter-textfile\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.147693 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.147623 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2b7070b6-7594-44c9-9d25-6ef3366255bf-node-exporter-tls\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.147693 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.147640 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b7070b6-7594-44c9-9d25-6ef3366255bf-sys\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.147693 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.147660 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2b7070b6-7594-44c9-9d25-6ef3366255bf-node-exporter-wtmp\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.147881 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.147725 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b7070b6-7594-44c9-9d25-6ef3366255bf-metrics-client-ca\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.147881 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.147763 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b7070b6-7594-44c9-9d25-6ef3366255bf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.248166 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.248136 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfc7f\" (UniqueName: \"kubernetes.io/projected/2b7070b6-7594-44c9-9d25-6ef3366255bf-kube-api-access-nfc7f\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.248551 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.248198 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2b7070b6-7594-44c9-9d25-6ef3366255bf-node-exporter-accelerators-collector-config\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.248551 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.248222 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2b7070b6-7594-44c9-9d25-6ef3366255bf-node-exporter-textfile\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.248551 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.248246 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2b7070b6-7594-44c9-9d25-6ef3366255bf-node-exporter-tls\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.248551 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.248264 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b7070b6-7594-44c9-9d25-6ef3366255bf-sys\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.248551 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.248295 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2b7070b6-7594-44c9-9d25-6ef3366255bf-node-exporter-wtmp\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.248551 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.248327 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b7070b6-7594-44c9-9d25-6ef3366255bf-metrics-client-ca\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.248551 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.248368 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b7070b6-7594-44c9-9d25-6ef3366255bf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.248551 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.248394 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b7070b6-7594-44c9-9d25-6ef3366255bf-sys\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.248551 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.248413 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2b7070b6-7594-44c9-9d25-6ef3366255bf-root\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.248551 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.248467 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2b7070b6-7594-44c9-9d25-6ef3366255bf-node-exporter-wtmp\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.248551 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.248513 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2b7070b6-7594-44c9-9d25-6ef3366255bf-root\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.249042 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.248830 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2b7070b6-7594-44c9-9d25-6ef3366255bf-node-exporter-textfile\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.249042 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.248932 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2b7070b6-7594-44c9-9d25-6ef3366255bf-node-exporter-accelerators-collector-config\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.249042 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.249006 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b7070b6-7594-44c9-9d25-6ef3366255bf-metrics-client-ca\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.250884 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.250862 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2b7070b6-7594-44c9-9d25-6ef3366255bf-node-exporter-tls\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.250989 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.250862 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b7070b6-7594-44c9-9d25-6ef3366255bf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.256695 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.256675 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfc7f\" (UniqueName: \"kubernetes.io/projected/2b7070b6-7594-44c9-9d25-6ef3366255bf-kube-api-access-nfc7f\") pod \"node-exporter-ssmlt\" (UID: \"2b7070b6-7594-44c9-9d25-6ef3366255bf\") " pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.363791 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:23.363761 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ssmlt" Jul 06 21:14:23.373599 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:14:23.373573 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7070b6_7594_44c9_9d25_6ef3366255bf.slice/crio-f841f722d7502c9f939f7570af2abf58e02d35a2a42311a9b40571cae49e7677 WatchSource:0}: Error finding container f841f722d7502c9f939f7570af2abf58e02d35a2a42311a9b40571cae49e7677: Status 404 returned error can't find the container with id f841f722d7502c9f939f7570af2abf58e02d35a2a42311a9b40571cae49e7677 Jul 06 21:14:24.128079 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.128040 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 21:14:24.132033 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.132012 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.134771 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.134746 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 21:14:24.134921 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.134770 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 21:14:24.134921 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.134828 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 21:14:24.135139 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.135114 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 21:14:24.135255 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.135203 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-v4fnz\"" Jul 06 21:14:24.135475 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.135446 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 21:14:24.135475 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.135459 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 21:14:24.135475 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.135463 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 21:14:24.135673 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.135511 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 21:14:24.135673 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.135518 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 21:14:24.147665 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.147639 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 21:14:24.256105 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.256085 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.256176 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.256115 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.256176 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.256133 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.256176 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.256152 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.256285 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.256192 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.256285 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.256221 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.256364 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.256293 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-config-volume\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.256364 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.256328 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.256433 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.256389 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-web-config\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.256466 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.256431 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.256466 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.256455 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.256525 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.256489 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-config-out\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.256525 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.256512 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsftn\" (UniqueName: \"kubernetes.io/projected/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-kube-api-access-vsftn\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.281637 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.281599 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ssmlt" event={"ID":"2b7070b6-7594-44c9-9d25-6ef3366255bf","Type":"ContainerStarted","Data":"f841f722d7502c9f939f7570af2abf58e02d35a2a42311a9b40571cae49e7677"} Jul 06 21:14:24.357594 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.357555 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-config-out\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.357594 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.357596 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsftn\" (UniqueName: \"kubernetes.io/projected/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-kube-api-access-vsftn\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.357809 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.357630 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.357809 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.357659 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.357809 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.357685 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.357809 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.357709 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.357809 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.357738 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.357809 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.357767 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.357809 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.357799 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-config-volume\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.358125 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.357824 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.358125 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:14:24.357847 2581 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 06 21:14:24.358125 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:14:24.357933 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-secret-alertmanager-main-tls podName:a3a65cf2-ed8f-4b79-b37d-0f669b835b7d nodeName:}" failed. No retries permitted until 2026-07-06 21:14:24.857911258 +0000 UTC m=+89.620824246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "a3a65cf2-ed8f-4b79-b37d-0f669b835b7d") : secret "alertmanager-main-tls" not found Jul 06 21:14:24.358125 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.358102 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.358125 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.357857 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-web-config\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.358409 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.358199 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.358409 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.358242 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.358862 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.358798 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.359768 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.359737 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.360629 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.360603 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-config-out\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.360767 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.360736 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.361302 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.361260 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-web-config\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.361421 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.361408 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.361595 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.361575 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.361663 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.361576 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.362307 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.362286 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.362816 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.362797 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-config-volume\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.368666 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.368646 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsftn\" (UniqueName: \"kubernetes.io/projected/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-kube-api-access-vsftn\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.862476 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:24.862434 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:24.862632 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:14:24.862566 2581 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 06 21:14:24.862632 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:14:24.862630 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-secret-alertmanager-main-tls podName:a3a65cf2-ed8f-4b79-b37d-0f669b835b7d nodeName:}" failed. No retries permitted until 2026-07-06 21:14:25.862615967 +0000 UTC m=+90.625528954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "a3a65cf2-ed8f-4b79-b37d-0f669b835b7d") : secret "alertmanager-main-tls" not found Jul 06 21:14:25.286940 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:25.286856 2581 generic.go:356] "Generic (PLEG): container finished" podID="2b7070b6-7594-44c9-9d25-6ef3366255bf" containerID="364a0dbc711031333687b091d9503096300e5c7cc4cd4da95d8f43cbf57eaf7e" exitCode=0 Jul 06 21:14:25.286940 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:25.286919 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ssmlt" event={"ID":"2b7070b6-7594-44c9-9d25-6ef3366255bf","Type":"ContainerDied","Data":"364a0dbc711031333687b091d9503096300e5c7cc4cd4da95d8f43cbf57eaf7e"} Jul 06 21:14:25.870261 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:25.870223 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:25.872938 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:25.872908 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a3a65cf2-ed8f-4b79-b37d-0f669b835b7d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:25.943063 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:25.943029 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:14:26.089744 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.089706 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 21:14:26.092492 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:14:26.092463 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a65cf2_ed8f_4b79_b37d_0f669b835b7d.slice/crio-379ba1a099eb68a7efc4a12f39a5392fcc16b05f6a6e0e63ac20b7f6d390793b WatchSource:0}: Error finding container 379ba1a099eb68a7efc4a12f39a5392fcc16b05f6a6e0e63ac20b7f6d390793b: Status 404 returned error can't find the container with id 379ba1a099eb68a7efc4a12f39a5392fcc16b05f6a6e0e63ac20b7f6d390793b Jul 06 21:14:26.152621 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.152543 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-886dfc74b-q78n8"] Jul 06 21:14:26.155631 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.155616 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.158087 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.158061 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 06 21:14:26.158230 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.158090 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 06 21:14:26.158230 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.158092 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 06 21:14:26.158230 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.158147 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 06 21:14:26.158230 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.158108 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-vlmoojch7vo3\"" Jul 06 21:14:26.158230 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.158092 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 06 21:14:26.158570 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.158550 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-b2vp7\"" Jul 06 21:14:26.169252 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.168894 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-886dfc74b-q78n8"] Jul 06 21:14:26.272607 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.272574 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b29fc180-58a8-477f-928b-3659cc343965-metrics-client-ca\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.272607 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.272620 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b29fc180-58a8-477f-928b-3659cc343965-secret-grpc-tls\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.272827 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.272656 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b29fc180-58a8-477f-928b-3659cc343965-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.272827 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.272680 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b29fc180-58a8-477f-928b-3659cc343965-secret-thanos-querier-tls\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.272827 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.272723 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b29fc180-58a8-477f-928b-3659cc343965-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.272827 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.272756 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bln9l\" (UniqueName: \"kubernetes.io/projected/b29fc180-58a8-477f-928b-3659cc343965-kube-api-access-bln9l\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.272827 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.272777 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b29fc180-58a8-477f-928b-3659cc343965-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.272990 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.272836 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b29fc180-58a8-477f-928b-3659cc343965-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.292656 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.292615 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ssmlt" event={"ID":"2b7070b6-7594-44c9-9d25-6ef3366255bf","Type":"ContainerStarted","Data":"f9ef55f974cc1b34ee99977947a122e8a2aad4733ef48a2caafe1216dd4b658a"} Jul 06 21:14:26.293074 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.292686 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ssmlt" event={"ID":"2b7070b6-7594-44c9-9d25-6ef3366255bf","Type":"ContainerStarted","Data":"47a8bd3fd1fa8b620e36a62522e795fc7072c0991b9fa2443838a0d22106248e"} Jul 06 21:14:26.293883 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.293855 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d","Type":"ContainerStarted","Data":"379ba1a099eb68a7efc4a12f39a5392fcc16b05f6a6e0e63ac20b7f6d390793b"} Jul 06 21:14:26.313142 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.313099 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ssmlt" podStartSLOduration=2.374173376 podStartE2EDuration="3.313085267s" totalImagesPullingTime="872.666087ms" totalInitContainerRuntime="66.245804ms" isStatefulPod=true podCreationTimestamp="2026-07-06 21:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:14:26.312185663 +0000 UTC m=+91.075098667" watchObservedRunningTime="2026-07-06 21:14:26.313085267 +0000 UTC m=+91.075998270" Jul 06 21:14:26.349653 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.349614 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:26.349805 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.349681 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:26.354427 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.354404 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:26.373505 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.373474 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b29fc180-58a8-477f-928b-3659cc343965-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.373661 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.373514 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b29fc180-58a8-477f-928b-3659cc343965-secret-thanos-querier-tls\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.373720 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.373660 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b29fc180-58a8-477f-928b-3659cc343965-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.373768 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.373746 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bln9l\" (UniqueName: \"kubernetes.io/projected/b29fc180-58a8-477f-928b-3659cc343965-kube-api-access-bln9l\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.373821 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.373798 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b29fc180-58a8-477f-928b-3659cc343965-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.373873 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.373837 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b29fc180-58a8-477f-928b-3659cc343965-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.373928 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.373888 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b29fc180-58a8-477f-928b-3659cc343965-metrics-client-ca\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.373991 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.373929 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b29fc180-58a8-477f-928b-3659cc343965-secret-grpc-tls\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.375000 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.374972 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b29fc180-58a8-477f-928b-3659cc343965-metrics-client-ca\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.376255 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.376227 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b29fc180-58a8-477f-928b-3659cc343965-secret-thanos-querier-tls\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.376787 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.376763 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b29fc180-58a8-477f-928b-3659cc343965-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.376873 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.376786 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b29fc180-58a8-477f-928b-3659cc343965-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.376873 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.376795 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b29fc180-58a8-477f-928b-3659cc343965-secret-grpc-tls\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.376873 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.376811 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b29fc180-58a8-477f-928b-3659cc343965-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.376972 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.376899 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b29fc180-58a8-477f-928b-3659cc343965-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.381314 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.381297 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bln9l\" (UniqueName: \"kubernetes.io/projected/b29fc180-58a8-477f-928b-3659cc343965-kube-api-access-bln9l\") pod \"thanos-querier-886dfc74b-q78n8\" (UID: \"b29fc180-58a8-477f-928b-3659cc343965\") " pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.465490 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.465409 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:26.603564 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:26.603538 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-886dfc74b-q78n8"] Jul 06 21:14:26.605980 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:14:26.605950 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29fc180_58a8_477f_928b_3659cc343965.slice/crio-82bbcdda6c8ebc8973c59ec5cbd38772922b41a34d246a75fe3d10ed9610f5a1 WatchSource:0}: Error finding container 82bbcdda6c8ebc8973c59ec5cbd38772922b41a34d246a75fe3d10ed9610f5a1: Status 404 returned error can't find the container with id 82bbcdda6c8ebc8973c59ec5cbd38772922b41a34d246a75fe3d10ed9610f5a1 Jul 06 21:14:27.299906 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:27.299830 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" event={"ID":"b29fc180-58a8-477f-928b-3659cc343965","Type":"ContainerStarted","Data":"82bbcdda6c8ebc8973c59ec5cbd38772922b41a34d246a75fe3d10ed9610f5a1"} Jul 06 21:14:27.306180 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:27.306154 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:14:28.305011 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:28.304981 2581 generic.go:356] "Generic (PLEG): container finished" podID="a3a65cf2-ed8f-4b79-b37d-0f669b835b7d" containerID="edf257df07542e439b381c5ab678770f2254c6db15a768cbf24332a66ade3070" exitCode=0 Jul 06 21:14:28.305446 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:28.305075 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d","Type":"ContainerDied","Data":"edf257df07542e439b381c5ab678770f2254c6db15a768cbf24332a66ade3070"} Jul 06 21:14:29.230688 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:29.230654 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:14:29.311012 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:29.310983 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" event={"ID":"b29fc180-58a8-477f-928b-3659cc343965","Type":"ContainerStarted","Data":"30558762e6bab414560e551943e3d642a35e79458cbcff779c8ee44a662a80d0"} Jul 06 21:14:29.311409 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:29.311050 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" event={"ID":"b29fc180-58a8-477f-928b-3659cc343965","Type":"ContainerStarted","Data":"d0e09f5bcf1762f9d7a0c61a3964fdb4d4d1c1ff86f666fd40e6953a97eddc61"} Jul 06 21:14:29.311409 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:29.311077 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" event={"ID":"b29fc180-58a8-477f-928b-3659cc343965","Type":"ContainerStarted","Data":"486cd6c7df97049a21f082946f5cb332519e0dc208be83d097a372ae04abbe0d"} Jul 06 21:14:30.247499 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.247423 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-559b687498-trz57"] Jul 06 21:14:30.249645 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.249630 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.261578 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.261549 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-559b687498-trz57"] Jul 06 21:14:30.317949 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.317909 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" event={"ID":"b29fc180-58a8-477f-928b-3659cc343965","Type":"ContainerStarted","Data":"601d8a74e4939dc02637129084ae91e77d581d5cf3a09095d53c658c81517dec"} Jul 06 21:14:30.318433 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.317986 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" event={"ID":"b29fc180-58a8-477f-928b-3659cc343965","Type":"ContainerStarted","Data":"d2f16e149e416408e4953ffec939c0378b41d8cf7a3adaece96689faa008d5f8"} Jul 06 21:14:30.318433 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.318022 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" event={"ID":"b29fc180-58a8-477f-928b-3659cc343965","Type":"ContainerStarted","Data":"e797e56c97a6ec1ccf9d12d0be3cabaecde2b323ae7ef4fd98f03f1675df8311"} Jul 06 21:14:30.318433 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.318067 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:30.321160 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.321137 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d","Type":"ContainerStarted","Data":"d49d66120026759b6c3817d812b17eab43b51169c6e2aa9481918707f5392708"} Jul 06 21:14:30.321289 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.321211 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d","Type":"ContainerStarted","Data":"b422cb467e88822e004548117eddea52050dd0108ea7ec899c493e36df85d99c"} Jul 06 21:14:30.321289 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.321250 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d","Type":"ContainerStarted","Data":"af4f9a15985aa9fc381dd30bcb8bf766ee58c56625cc896b729d4ef251692c6a"} Jul 06 21:14:30.321289 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.321284 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d","Type":"ContainerStarted","Data":"839a314b8aa6baf00f8a9f3c2d89d6b940ff9a31262110e506c50b143b1f7b3b"} Jul 06 21:14:30.321495 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.321324 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d","Type":"ContainerStarted","Data":"1f783a302123e2f89a615d73720da12f42b450a5b682f868fc049373ac6fca1d"} Jul 06 21:14:30.321495 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.321378 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3a65cf2-ed8f-4b79-b37d-0f669b835b7d","Type":"ContainerStarted","Data":"211055a7a79fb0d9a86ab5e8bd760425d5ed6f3afaecfc8b24eccc144aefed7a"} Jul 06 21:14:30.341031 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.340985 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" podStartSLOduration=1.811717765 podStartE2EDuration="4.340969968s" totalImagesPullingTime="2.529252203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:14:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:14:30.339529799 +0000 UTC m=+95.102442803" watchObservedRunningTime="2026-07-06 21:14:30.340969968 +0000 UTC m=+95.103882972" Jul 06 21:14:30.370932 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.370878 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.79173661 podStartE2EDuration="6.370861117s" totalImagesPullingTime="2.557082161s" totalInitContainerRuntime="22.042346ms" isStatefulPod=false podCreationTimestamp="2026-07-06 21:14:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:14:30.369029154 +0000 UTC m=+95.131942159" watchObservedRunningTime="2026-07-06 21:14:30.370861117 +0000 UTC m=+95.133774120" Jul 06 21:14:30.409248 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.409215 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-console-oauth-config\") pod \"console-559b687498-trz57\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.409446 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.409299 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjcbq\" (UniqueName: \"kubernetes.io/projected/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-kube-api-access-qjcbq\") pod \"console-559b687498-trz57\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.409446 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.409394 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-trusted-ca-bundle\") pod \"console-559b687498-trz57\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.409577 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.409504 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-service-ca\") pod \"console-559b687498-trz57\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.409632 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.409592 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-oauth-serving-cert\") pod \"console-559b687498-trz57\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.409938 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.409902 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-console-serving-cert\") pod \"console-559b687498-trz57\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.410063 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.409970 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-console-config\") pod \"console-559b687498-trz57\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.510957 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.510880 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-trusted-ca-bundle\") pod \"console-559b687498-trz57\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.510957 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.510920 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-service-ca\") pod \"console-559b687498-trz57\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.510957 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.510942 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-oauth-serving-cert\") pod \"console-559b687498-trz57\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.511241 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.510970 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-console-serving-cert\") pod \"console-559b687498-trz57\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.511241 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.510987 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-console-config\") pod \"console-559b687498-trz57\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.511241 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.511015 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-console-oauth-config\") pod \"console-559b687498-trz57\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.511241 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.511051 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjcbq\" (UniqueName: \"kubernetes.io/projected/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-kube-api-access-qjcbq\") pod \"console-559b687498-trz57\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.511865 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.511839 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-oauth-serving-cert\") pod \"console-559b687498-trz57\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.511990 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.511884 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-trusted-ca-bundle\") pod \"console-559b687498-trz57\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.511990 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.511893 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-service-ca\") pod \"console-559b687498-trz57\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.512093 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.512033 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-console-config\") pod \"console-559b687498-trz57\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.513656 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.513627 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-console-serving-cert\") pod \"console-559b687498-trz57\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.513656 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.513647 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-console-oauth-config\") pod \"console-559b687498-trz57\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.519706 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.519683 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjcbq\" (UniqueName: \"kubernetes.io/projected/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-kube-api-access-qjcbq\") pod \"console-559b687498-trz57\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.558893 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.558852 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:30.700076 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:30.700050 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-559b687498-trz57"] Jul 06 21:14:30.702213 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:14:30.702188 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9fa581_2f55_4b56_ac19_d45ac4ebf58d.slice/crio-26b086f0084e27c6a0c0d155b956075edfab4d3fa80bba8808ec357f1c78c237 WatchSource:0}: Error finding container 26b086f0084e27c6a0c0d155b956075edfab4d3fa80bba8808ec357f1c78c237: Status 404 returned error can't find the container with id 26b086f0084e27c6a0c0d155b956075edfab4d3fa80bba8808ec357f1c78c237 Jul 06 21:14:31.327062 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:31.327023 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559b687498-trz57" event={"ID":"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d","Type":"ContainerStarted","Data":"c819ab9c206d8f3aae773b6e718244de4fb1e51e1cbcdf446332356c2d84cc86"} Jul 06 21:14:31.327464 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:31.327116 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559b687498-trz57" event={"ID":"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d","Type":"ContainerStarted","Data":"26b086f0084e27c6a0c0d155b956075edfab4d3fa80bba8808ec357f1c78c237"} Jul 06 21:14:31.347223 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:31.347175 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-559b687498-trz57" podStartSLOduration=1.347158683 podStartE2EDuration="1.347158683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:14:31.346594916 +0000 UTC m=+96.109507934" watchObservedRunningTime="2026-07-06 21:14:31.347158683 +0000 UTC m=+96.110071688" Jul 06 21:14:33.187518 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:33.187484 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wjc92" Jul 06 21:14:34.242821 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.242778 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" podUID="da180245-4bba-4cbd-8bb6-6e370897c52d" containerName="registry" containerID="cri-o://5d142067fc2755b68870cef59bb14055d42b065df54b3c011db5e5f01a5e7b72" gracePeriod=30 Jul 06 21:14:34.474463 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.474437 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:14:34.650286 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.650241 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da180245-4bba-4cbd-8bb6-6e370897c52d-installation-pull-secrets\") pod \"da180245-4bba-4cbd-8bb6-6e370897c52d\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " Jul 06 21:14:34.650468 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.650298 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da180245-4bba-4cbd-8bb6-6e370897c52d-trusted-ca\") pod \"da180245-4bba-4cbd-8bb6-6e370897c52d\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " Jul 06 21:14:34.650526 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.650504 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-certificates\") pod \"da180245-4bba-4cbd-8bb6-6e370897c52d\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " Jul 06 21:14:34.650574 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.650560 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56kbh\" (UniqueName: \"kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-kube-api-access-56kbh\") pod \"da180245-4bba-4cbd-8bb6-6e370897c52d\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " Jul 06 21:14:34.650624 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.650601 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls\") pod \"da180245-4bba-4cbd-8bb6-6e370897c52d\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " Jul 06 21:14:34.650669 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.650630 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da180245-4bba-4cbd-8bb6-6e370897c52d-image-registry-private-configuration\") pod \"da180245-4bba-4cbd-8bb6-6e370897c52d\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " Jul 06 21:14:34.650704 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.650675 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-bound-sa-token\") pod \"da180245-4bba-4cbd-8bb6-6e370897c52d\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " Jul 06 21:14:34.650755 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.650717 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da180245-4bba-4cbd-8bb6-6e370897c52d-ca-trust-extracted\") pod \"da180245-4bba-4cbd-8bb6-6e370897c52d\" (UID: \"da180245-4bba-4cbd-8bb6-6e370897c52d\") " Jul 06 21:14:34.650868 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.650843 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da180245-4bba-4cbd-8bb6-6e370897c52d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "da180245-4bba-4cbd-8bb6-6e370897c52d" (UID: "da180245-4bba-4cbd-8bb6-6e370897c52d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:14:34.651085 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.650946 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "da180245-4bba-4cbd-8bb6-6e370897c52d" (UID: "da180245-4bba-4cbd-8bb6-6e370897c52d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:14:34.651085 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.651085 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da180245-4bba-4cbd-8bb6-6e370897c52d-trusted-ca\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:14:34.651267 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.651104 2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-certificates\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:14:34.653168 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.653139 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da180245-4bba-4cbd-8bb6-6e370897c52d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "da180245-4bba-4cbd-8bb6-6e370897c52d" (UID: "da180245-4bba-4cbd-8bb6-6e370897c52d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:14:34.653286 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.653141 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-kube-api-access-56kbh" (OuterVolumeSpecName: "kube-api-access-56kbh") pod "da180245-4bba-4cbd-8bb6-6e370897c52d" (UID: "da180245-4bba-4cbd-8bb6-6e370897c52d"). InnerVolumeSpecName "kube-api-access-56kbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:14:34.653286 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.653224 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "da180245-4bba-4cbd-8bb6-6e370897c52d" (UID: "da180245-4bba-4cbd-8bb6-6e370897c52d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:14:34.653286 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.653257 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da180245-4bba-4cbd-8bb6-6e370897c52d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "da180245-4bba-4cbd-8bb6-6e370897c52d" (UID: "da180245-4bba-4cbd-8bb6-6e370897c52d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:14:34.653494 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.653468 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "da180245-4bba-4cbd-8bb6-6e370897c52d" (UID: "da180245-4bba-4cbd-8bb6-6e370897c52d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:14:34.658993 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.658970 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da180245-4bba-4cbd-8bb6-6e370897c52d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "da180245-4bba-4cbd-8bb6-6e370897c52d" (UID: "da180245-4bba-4cbd-8bb6-6e370897c52d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:14:34.751958 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.751922 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56kbh\" (UniqueName: \"kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-kube-api-access-56kbh\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:14:34.751958 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.751955 2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-registry-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:14:34.751958 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.751966 2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da180245-4bba-4cbd-8bb6-6e370897c52d-image-registry-private-configuration\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:14:34.752185 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.751978 2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da180245-4bba-4cbd-8bb6-6e370897c52d-bound-sa-token\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:14:34.752185 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.751986 2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da180245-4bba-4cbd-8bb6-6e370897c52d-ca-trust-extracted\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:14:34.752185 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:34.751994 2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da180245-4bba-4cbd-8bb6-6e370897c52d-installation-pull-secrets\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:14:35.343871 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:35.343835 2581 generic.go:356] "Generic (PLEG): container finished" podID="da180245-4bba-4cbd-8bb6-6e370897c52d" containerID="5d142067fc2755b68870cef59bb14055d42b065df54b3c011db5e5f01a5e7b72" exitCode=0 Jul 06 21:14:35.344248 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:35.343898 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" Jul 06 21:14:35.344248 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:35.343922 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" event={"ID":"da180245-4bba-4cbd-8bb6-6e370897c52d","Type":"ContainerDied","Data":"5d142067fc2755b68870cef59bb14055d42b065df54b3c011db5e5f01a5e7b72"} Jul 06 21:14:35.344248 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:35.343976 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85bbcdb746-lph2c" event={"ID":"da180245-4bba-4cbd-8bb6-6e370897c52d","Type":"ContainerDied","Data":"8e2aeeceda87f74c51ac8699fbf49a1e4e5d57cfe872a6a0f7d1dfd41da04040"} Jul 06 21:14:35.344248 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:35.343997 2581 scope.go:117] "RemoveContainer" containerID="5d142067fc2755b68870cef59bb14055d42b065df54b3c011db5e5f01a5e7b72" Jul 06 21:14:35.353716 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:35.353689 2581 scope.go:117] "RemoveContainer" containerID="5d142067fc2755b68870cef59bb14055d42b065df54b3c011db5e5f01a5e7b72" Jul 06 21:14:35.353935 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:14:35.353915 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d142067fc2755b68870cef59bb14055d42b065df54b3c011db5e5f01a5e7b72\": container with ID starting with 5d142067fc2755b68870cef59bb14055d42b065df54b3c011db5e5f01a5e7b72 not found: ID does not exist" containerID="5d142067fc2755b68870cef59bb14055d42b065df54b3c011db5e5f01a5e7b72" Jul 06 21:14:35.353982 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:35.353943 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d142067fc2755b68870cef59bb14055d42b065df54b3c011db5e5f01a5e7b72"} err="failed to get container status \"5d142067fc2755b68870cef59bb14055d42b065df54b3c011db5e5f01a5e7b72\": rpc error: code = NotFound desc = could not find container \"5d142067fc2755b68870cef59bb14055d42b065df54b3c011db5e5f01a5e7b72\": container with ID starting with 5d142067fc2755b68870cef59bb14055d42b065df54b3c011db5e5f01a5e7b72 not found: ID does not exist" Jul 06 21:14:35.365299 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:35.365267 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85bbcdb746-lph2c"] Jul 06 21:14:35.369053 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:35.369032 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-85bbcdb746-lph2c"] Jul 06 21:14:35.785649 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:35.785617 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da180245-4bba-4cbd-8bb6-6e370897c52d" path="/var/lib/kubelet/pods/da180245-4bba-4cbd-8bb6-6e370897c52d/volumes" Jul 06 21:14:36.335258 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:36.335233 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-886dfc74b-q78n8" Jul 06 21:14:40.559766 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:40.559729 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:40.560162 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:40.559799 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:40.564688 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:40.564662 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:41.370606 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:41.370575 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-559b687498-trz57" Jul 06 21:14:41.422038 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:41.422005 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74c76db575-thdzm"] Jul 06 21:14:51.405484 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:51.405448 2581 generic.go:356] "Generic (PLEG): container finished" podID="786193aa-77dd-42c4-8944-0aadc3a864f0" containerID="0b41b6888222a3231d3006394b5371be70ea8a576292a5a950b9bee1bee6b54e" exitCode=0 Jul 06 21:14:51.405928 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:51.405524 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9qb5m" event={"ID":"786193aa-77dd-42c4-8944-0aadc3a864f0","Type":"ContainerDied","Data":"0b41b6888222a3231d3006394b5371be70ea8a576292a5a950b9bee1bee6b54e"} Jul 06 21:14:51.405928 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:51.405851 2581 scope.go:117] "RemoveContainer" containerID="0b41b6888222a3231d3006394b5371be70ea8a576292a5a950b9bee1bee6b54e" Jul 06 21:14:52.411435 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:14:52.411401 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9qb5m" event={"ID":"786193aa-77dd-42c4-8944-0aadc3a864f0","Type":"ContainerStarted","Data":"6528f99f74f55d7fe46d3cfc109636c1c2feff1741c26490aaa4570cf36fbb76"} Jul 06 21:15:06.442549 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.442487 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-74c76db575-thdzm" podUID="7aec20eb-fd13-4135-95d0-ddea26dd1001" containerName="console" containerID="cri-o://146ac90790f5c7f277f3e4001d794cd489a1b4a26d32104c5dec4a8240cee50e" gracePeriod=15 Jul 06 21:15:06.677648 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.677626 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:15:06.799131 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.799014 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7aec20eb-fd13-4135-95d0-ddea26dd1001-console-serving-cert\") pod \"7aec20eb-fd13-4135-95d0-ddea26dd1001\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " Jul 06 21:15:06.799131 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.799098 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-trusted-ca-bundle\") pod \"7aec20eb-fd13-4135-95d0-ddea26dd1001\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " Jul 06 21:15:06.799403 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.799149 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94pvn\" (UniqueName: \"kubernetes.io/projected/7aec20eb-fd13-4135-95d0-ddea26dd1001-kube-api-access-94pvn\") pod \"7aec20eb-fd13-4135-95d0-ddea26dd1001\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " Jul 06 21:15:06.799403 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.799181 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-oauth-serving-cert\") pod \"7aec20eb-fd13-4135-95d0-ddea26dd1001\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " Jul 06 21:15:06.799403 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.799197 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-console-config\") pod \"7aec20eb-fd13-4135-95d0-ddea26dd1001\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " Jul 06 21:15:06.799403 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.799225 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-service-ca\") pod \"7aec20eb-fd13-4135-95d0-ddea26dd1001\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " Jul 06 21:15:06.799403 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.799245 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7aec20eb-fd13-4135-95d0-ddea26dd1001-console-oauth-config\") pod \"7aec20eb-fd13-4135-95d0-ddea26dd1001\" (UID: \"7aec20eb-fd13-4135-95d0-ddea26dd1001\") " Jul 06 21:15:06.799744 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.799699 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7aec20eb-fd13-4135-95d0-ddea26dd1001" (UID: "7aec20eb-fd13-4135-95d0-ddea26dd1001"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:15:06.799744 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.799723 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-console-config" (OuterVolumeSpecName: "console-config") pod "7aec20eb-fd13-4135-95d0-ddea26dd1001" (UID: "7aec20eb-fd13-4135-95d0-ddea26dd1001"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:15:06.799862 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.799733 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7aec20eb-fd13-4135-95d0-ddea26dd1001" (UID: "7aec20eb-fd13-4135-95d0-ddea26dd1001"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:15:06.799862 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.799845 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-service-ca" (OuterVolumeSpecName: "service-ca") pod "7aec20eb-fd13-4135-95d0-ddea26dd1001" (UID: "7aec20eb-fd13-4135-95d0-ddea26dd1001"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:15:06.801430 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.801402 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aec20eb-fd13-4135-95d0-ddea26dd1001-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7aec20eb-fd13-4135-95d0-ddea26dd1001" (UID: "7aec20eb-fd13-4135-95d0-ddea26dd1001"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:15:06.801538 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.801454 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aec20eb-fd13-4135-95d0-ddea26dd1001-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7aec20eb-fd13-4135-95d0-ddea26dd1001" (UID: "7aec20eb-fd13-4135-95d0-ddea26dd1001"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:15:06.801587 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.801526 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aec20eb-fd13-4135-95d0-ddea26dd1001-kube-api-access-94pvn" (OuterVolumeSpecName: "kube-api-access-94pvn") pod "7aec20eb-fd13-4135-95d0-ddea26dd1001" (UID: "7aec20eb-fd13-4135-95d0-ddea26dd1001"). InnerVolumeSpecName "kube-api-access-94pvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:15:06.900596 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.900557 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-trusted-ca-bundle\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:15:06.900596 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.900588 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94pvn\" (UniqueName: \"kubernetes.io/projected/7aec20eb-fd13-4135-95d0-ddea26dd1001-kube-api-access-94pvn\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:15:06.900596 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.900602 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-oauth-serving-cert\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:15:06.900829 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.900614 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-console-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:15:06.900829 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.900625 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7aec20eb-fd13-4135-95d0-ddea26dd1001-service-ca\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:15:06.900829 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.900636 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7aec20eb-fd13-4135-95d0-ddea26dd1001-console-oauth-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:15:06.900829 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:06.900646 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7aec20eb-fd13-4135-95d0-ddea26dd1001-console-serving-cert\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:15:07.469069 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:07.469036 2581 generic.go:356] "Generic (PLEG): container finished" podID="7aec20eb-fd13-4135-95d0-ddea26dd1001" containerID="146ac90790f5c7f277f3e4001d794cd489a1b4a26d32104c5dec4a8240cee50e" exitCode=0 Jul 06 21:15:07.469529 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:07.469083 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c76db575-thdzm" Jul 06 21:15:07.469529 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:07.469112 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c76db575-thdzm" event={"ID":"7aec20eb-fd13-4135-95d0-ddea26dd1001","Type":"ContainerDied","Data":"146ac90790f5c7f277f3e4001d794cd489a1b4a26d32104c5dec4a8240cee50e"} Jul 06 21:15:07.469529 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:07.469152 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c76db575-thdzm" event={"ID":"7aec20eb-fd13-4135-95d0-ddea26dd1001","Type":"ContainerDied","Data":"be06e14847c9629954135c33001efdaaaa128ecdf2d50f963a238195f5ad3d21"} Jul 06 21:15:07.469529 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:07.469171 2581 scope.go:117] "RemoveContainer" containerID="146ac90790f5c7f277f3e4001d794cd489a1b4a26d32104c5dec4a8240cee50e" Jul 06 21:15:07.477749 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:07.477731 2581 scope.go:117] "RemoveContainer" containerID="146ac90790f5c7f277f3e4001d794cd489a1b4a26d32104c5dec4a8240cee50e" Jul 06 21:15:07.477989 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:15:07.477970 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"146ac90790f5c7f277f3e4001d794cd489a1b4a26d32104c5dec4a8240cee50e\": container with ID starting with 146ac90790f5c7f277f3e4001d794cd489a1b4a26d32104c5dec4a8240cee50e not found: ID does not exist" containerID="146ac90790f5c7f277f3e4001d794cd489a1b4a26d32104c5dec4a8240cee50e" Jul 06 21:15:07.478052 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:07.478004 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"146ac90790f5c7f277f3e4001d794cd489a1b4a26d32104c5dec4a8240cee50e"} err="failed to get container status \"146ac90790f5c7f277f3e4001d794cd489a1b4a26d32104c5dec4a8240cee50e\": rpc error: code = NotFound desc = could not find container \"146ac90790f5c7f277f3e4001d794cd489a1b4a26d32104c5dec4a8240cee50e\": container with ID starting with 146ac90790f5c7f277f3e4001d794cd489a1b4a26d32104c5dec4a8240cee50e not found: ID does not exist" Jul 06 21:15:07.489608 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:07.489585 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74c76db575-thdzm"] Jul 06 21:15:07.492636 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:07.492612 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74c76db575-thdzm"] Jul 06 21:15:07.785417 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:07.785320 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aec20eb-fd13-4135-95d0-ddea26dd1001" path="/var/lib/kubelet/pods/7aec20eb-fd13-4135-95d0-ddea26dd1001/volumes" Jul 06 21:15:11.485731 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:11.485697 2581 generic.go:356] "Generic (PLEG): container finished" podID="2a156ee3-56b6-45e1-80de-debe10e30cca" containerID="bf571484cc00c9454462354799ce0637463ddccbbadb69e956202230a43ab5e7" exitCode=0 Jul 06 21:15:11.486113 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:11.485753 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" event={"ID":"2a156ee3-56b6-45e1-80de-debe10e30cca","Type":"ContainerDied","Data":"bf571484cc00c9454462354799ce0637463ddccbbadb69e956202230a43ab5e7"} Jul 06 21:15:11.486113 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:11.486089 2581 scope.go:117] "RemoveContainer" containerID="bf571484cc00c9454462354799ce0637463ddccbbadb69e956202230a43ab5e7" Jul 06 21:15:12.490874 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:12.490839 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-cnsh7" event={"ID":"2a156ee3-56b6-45e1-80de-debe10e30cca","Type":"ContainerStarted","Data":"56058428d5e988d9443c9efff281ba20501e2d11a75c76bf53192bc25ee9ca6f"} Jul 06 21:15:16.506148 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:16.506116 2581 generic.go:356] "Generic (PLEG): container finished" podID="b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be" containerID="c96e15ae2857b8aee88aa31a76397a63a7c3784cf723b00c55870a7492e00b12" exitCode=0 Jul 06 21:15:16.506570 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:16.506199 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2tsvl" event={"ID":"b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be","Type":"ContainerDied","Data":"c96e15ae2857b8aee88aa31a76397a63a7c3784cf723b00c55870a7492e00b12"} Jul 06 21:15:16.506633 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:16.506574 2581 scope.go:117] "RemoveContainer" containerID="c96e15ae2857b8aee88aa31a76397a63a7c3784cf723b00c55870a7492e00b12" Jul 06 21:15:17.511692 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:17.511651 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2tsvl" event={"ID":"b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be","Type":"ContainerStarted","Data":"4a4b16f70cd527e731f001151f5c96cb8fcfca9acb855fabba72dfec5e7aebd5"} Jul 06 21:15:47.354273 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.354239 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh"] Jul 06 21:15:47.354754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.354569 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da180245-4bba-4cbd-8bb6-6e370897c52d" containerName="registry" Jul 06 21:15:47.354754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.354581 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="da180245-4bba-4cbd-8bb6-6e370897c52d" containerName="registry" Jul 06 21:15:47.354754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.354601 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aec20eb-fd13-4135-95d0-ddea26dd1001" containerName="console" Jul 06 21:15:47.354754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.354606 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aec20eb-fd13-4135-95d0-ddea26dd1001" containerName="console" Jul 06 21:15:47.354754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.354670 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="da180245-4bba-4cbd-8bb6-6e370897c52d" containerName="registry" Jul 06 21:15:47.354754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.354679 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aec20eb-fd13-4135-95d0-ddea26dd1001" containerName="console" Jul 06 21:15:47.360914 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.360880 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.364228 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.364201 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 06 21:15:47.364228 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.364220 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 06 21:15:47.364451 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.364201 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 06 21:15:47.364451 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.364201 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 06 21:15:47.364451 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.364395 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 06 21:15:47.364577 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.364547 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-dqsqt\"" Jul 06 21:15:47.369123 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.369105 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 06 21:15:47.373182 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.373162 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh"] Jul 06 21:15:47.438472 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.438422 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.438472 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.438482 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-serving-certs-ca-bundle\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.438725 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.438536 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-telemeter-client-tls\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.438725 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.438574 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-federate-client-tls\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.438725 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.438605 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-secret-telemeter-client\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.438725 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.438634 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.438725 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.438686 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-metrics-client-ca\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.438898 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.438728 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvhgv\" (UniqueName: \"kubernetes.io/projected/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-kube-api-access-jvhgv\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.539502 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.539448 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.539502 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.539506 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-serving-certs-ca-bundle\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.539773 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.539545 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-telemeter-client-tls\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.539773 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.539574 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-federate-client-tls\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.539773 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.539610 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-secret-telemeter-client\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.539773 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.539642 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.539773 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.539674 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-metrics-client-ca\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.539773 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.539713 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvhgv\" (UniqueName: \"kubernetes.io/projected/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-kube-api-access-jvhgv\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.540501 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.540474 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-serving-certs-ca-bundle\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.540628 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.540522 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-metrics-client-ca\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.540628 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.540602 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.542625 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.542596 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.542719 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.542650 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-secret-telemeter-client\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.542833 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.542812 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-federate-client-tls\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.542869 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.542816 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-telemeter-client-tls\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.548883 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.548859 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvhgv\" (UniqueName: \"kubernetes.io/projected/43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1-kube-api-access-jvhgv\") pod \"telemeter-client-6f4d7fcb8b-7pjxh\" (UID: \"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1\") " pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.670572 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.670474 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" Jul 06 21:15:47.801697 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:47.801675 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh"] Jul 06 21:15:47.804281 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:15:47.804259 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d4bdd3_6182_4ada_90d6_4bdfcfff4eb1.slice/crio-9f5f3a50a000bffbbc102c13747f51227a2e962fac605b1798113830753b8f8f WatchSource:0}: Error finding container 9f5f3a50a000bffbbc102c13747f51227a2e962fac605b1798113830753b8f8f: Status 404 returned error can't find the container with id 9f5f3a50a000bffbbc102c13747f51227a2e962fac605b1798113830753b8f8f Jul 06 21:15:48.635330 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:48.635293 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" event={"ID":"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1","Type":"ContainerStarted","Data":"9f5f3a50a000bffbbc102c13747f51227a2e962fac605b1798113830753b8f8f"} Jul 06 21:15:50.644949 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:50.644914 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" event={"ID":"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1","Type":"ContainerStarted","Data":"c4e747fd731ae18952eb79d03f903fca273e99d8fad574cd47f5479782ee310c"} Jul 06 21:15:50.645397 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:50.644969 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" event={"ID":"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1","Type":"ContainerStarted","Data":"318258b071b181885f6a8d4723c007072a62cab5c5b852035e829f7733eacd76"} Jul 06 21:15:50.645397 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:50.644988 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" event={"ID":"43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1","Type":"ContainerStarted","Data":"0c1e0664c57c9054c54bbd4c35239e6240337343101dc31960f4229eb3702387"} Jul 06 21:15:50.668166 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:50.668121 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6f4d7fcb8b-7pjxh" podStartSLOduration=1.785831318 podStartE2EDuration="3.668106281s" totalImagesPullingTime="1.882274963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:15:50.667738303 +0000 UTC m=+175.430651310" watchObservedRunningTime="2026-07-06 21:15:50.668106281 +0000 UTC m=+175.431019285" Jul 06 21:15:51.502674 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.502637 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7449c7544b-vgfgs"] Jul 06 21:15:51.506214 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.506196 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.523200 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.523163 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7449c7544b-vgfgs"] Jul 06 21:15:51.574931 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.574890 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-trusted-ca-bundle\") pod \"console-7449c7544b-vgfgs\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.575126 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.574939 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-console-config\") pod \"console-7449c7544b-vgfgs\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.575126 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.575024 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-service-ca\") pod \"console-7449c7544b-vgfgs\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.575126 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.575074 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-oauth-serving-cert\") pod \"console-7449c7544b-vgfgs\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.575257 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.575120 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-574br\" (UniqueName: \"kubernetes.io/projected/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-kube-api-access-574br\") pod \"console-7449c7544b-vgfgs\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.575257 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.575154 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-console-oauth-config\") pod \"console-7449c7544b-vgfgs\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.575257 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.575192 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-console-serving-cert\") pod \"console-7449c7544b-vgfgs\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.676508 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.676476 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-service-ca\") pod \"console-7449c7544b-vgfgs\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.676976 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.676518 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-oauth-serving-cert\") pod \"console-7449c7544b-vgfgs\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.676976 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.676534 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-574br\" (UniqueName: \"kubernetes.io/projected/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-kube-api-access-574br\") pod \"console-7449c7544b-vgfgs\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.676976 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.676557 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-console-oauth-config\") pod \"console-7449c7544b-vgfgs\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.676976 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.676584 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-console-serving-cert\") pod \"console-7449c7544b-vgfgs\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.676976 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.676787 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-trusted-ca-bundle\") pod \"console-7449c7544b-vgfgs\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.676976 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.676865 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-console-config\") pod \"console-7449c7544b-vgfgs\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.677406 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.677309 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-service-ca\") pod \"console-7449c7544b-vgfgs\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.677458 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.677432 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-oauth-serving-cert\") pod \"console-7449c7544b-vgfgs\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.677644 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.677622 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-console-config\") pod \"console-7449c7544b-vgfgs\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.677849 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.677830 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-trusted-ca-bundle\") pod \"console-7449c7544b-vgfgs\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.679497 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.679467 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-console-serving-cert\") pod \"console-7449c7544b-vgfgs\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.679497 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.679482 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-console-oauth-config\") pod \"console-7449c7544b-vgfgs\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.684488 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.684472 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-574br\" (UniqueName: \"kubernetes.io/projected/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-kube-api-access-574br\") pod \"console-7449c7544b-vgfgs\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.815698 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.815607 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:15:51.943829 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:51.943796 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7449c7544b-vgfgs"] Jul 06 21:15:51.946911 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:15:51.946884 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8c24c7_b20e_4548_a9a8_92d6d0d83004.slice/crio-0114880db9b7b561f485bf55b0f2b6707a6f476cdf6de018cf66b0fba278e216 WatchSource:0}: Error finding container 0114880db9b7b561f485bf55b0f2b6707a6f476cdf6de018cf66b0fba278e216: Status 404 returned error can't find the container with id 0114880db9b7b561f485bf55b0f2b6707a6f476cdf6de018cf66b0fba278e216 Jul 06 21:15:52.654429 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:52.654395 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7449c7544b-vgfgs" event={"ID":"bb8c24c7-b20e-4548-a9a8-92d6d0d83004","Type":"ContainerStarted","Data":"fb54d2b87661b86deb795f97137413c458e03f969b7e80ee1fbdffee5837a36b"} Jul 06 21:15:52.654609 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:52.654443 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7449c7544b-vgfgs" event={"ID":"bb8c24c7-b20e-4548-a9a8-92d6d0d83004","Type":"ContainerStarted","Data":"0114880db9b7b561f485bf55b0f2b6707a6f476cdf6de018cf66b0fba278e216"} Jul 06 21:15:52.672692 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:52.672648 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7449c7544b-vgfgs" podStartSLOduration=1.67263296 podStartE2EDuration="1.67263296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:15:52.670867505 +0000 UTC m=+177.433780509" watchObservedRunningTime="2026-07-06 21:15:52.67263296 +0000 UTC m=+177.435545964" Jul 06 21:15:57.549149 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.549117 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7449c7544b-vgfgs"] Jul 06 21:15:57.582849 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.582818 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66b65fc796-l2drm"] Jul 06 21:15:57.586005 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.585987 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:57.597672 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.597649 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66b65fc796-l2drm"] Jul 06 21:15:57.626521 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.626495 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-service-ca\") pod \"console-66b65fc796-l2drm\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:57.626676 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.626532 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-trusted-ca-bundle\") pod \"console-66b65fc796-l2drm\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:57.626676 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.626563 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-oauth-serving-cert\") pod \"console-66b65fc796-l2drm\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:57.626796 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.626680 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-console-serving-cert\") pod \"console-66b65fc796-l2drm\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:57.626796 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.626713 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l56lk\" (UniqueName: \"kubernetes.io/projected/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-kube-api-access-l56lk\") pod \"console-66b65fc796-l2drm\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:57.626796 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.626744 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-console-config\") pod \"console-66b65fc796-l2drm\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:57.626796 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.626773 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-console-oauth-config\") pod \"console-66b65fc796-l2drm\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:57.727403 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.727367 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-console-serving-cert\") pod \"console-66b65fc796-l2drm\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:57.727403 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.727404 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l56lk\" (UniqueName: \"kubernetes.io/projected/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-kube-api-access-l56lk\") pod \"console-66b65fc796-l2drm\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:57.727647 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.727429 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-console-config\") pod \"console-66b65fc796-l2drm\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:57.727647 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.727451 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-console-oauth-config\") pod \"console-66b65fc796-l2drm\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:57.727647 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.727500 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-service-ca\") pod \"console-66b65fc796-l2drm\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:57.727647 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.727535 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-trusted-ca-bundle\") pod \"console-66b65fc796-l2drm\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:57.727647 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.727580 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-oauth-serving-cert\") pod \"console-66b65fc796-l2drm\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:57.728268 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.728237 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-console-config\") pod \"console-66b65fc796-l2drm\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:57.728268 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.728256 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-service-ca\") pod \"console-66b65fc796-l2drm\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:57.728268 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.728266 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-oauth-serving-cert\") pod \"console-66b65fc796-l2drm\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:57.728511 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.728377 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-trusted-ca-bundle\") pod \"console-66b65fc796-l2drm\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:57.729866 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.729837 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-console-oauth-config\") pod \"console-66b65fc796-l2drm\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:57.730077 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.730058 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-console-serving-cert\") pod \"console-66b65fc796-l2drm\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:57.738713 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.738695 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l56lk\" (UniqueName: \"kubernetes.io/projected/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-kube-api-access-l56lk\") pod \"console-66b65fc796-l2drm\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:57.895133 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:57.895095 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:15:58.020707 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:58.020674 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66b65fc796-l2drm"] Jul 06 21:15:58.023620 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:15:58.023595 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0c50e5_e18e_4a6d_b2c5_fb9d624a9b3e.slice/crio-2d8d10d9f37092c8e3e07444c14fa31b79f6d9211f6bf4788592ef64d4ab340f WatchSource:0}: Error finding container 2d8d10d9f37092c8e3e07444c14fa31b79f6d9211f6bf4788592ef64d4ab340f: Status 404 returned error can't find the container with id 2d8d10d9f37092c8e3e07444c14fa31b79f6d9211f6bf4788592ef64d4ab340f Jul 06 21:15:58.679895 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:58.679860 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b65fc796-l2drm" event={"ID":"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e","Type":"ContainerStarted","Data":"23f351253e39e3f809cbcfbba85be10a2f6045f975a3563ff38e810bc4bf9e56"} Jul 06 21:15:58.680278 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:58.679909 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b65fc796-l2drm" event={"ID":"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e","Type":"ContainerStarted","Data":"2d8d10d9f37092c8e3e07444c14fa31b79f6d9211f6bf4788592ef64d4ab340f"} Jul 06 21:15:58.699835 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:15:58.699788 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66b65fc796-l2drm" podStartSLOduration=1.699771683 podStartE2EDuration="1.699771683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:15:58.697627717 +0000 UTC m=+183.460540721" watchObservedRunningTime="2026-07-06 21:15:58.699771683 +0000 UTC m=+183.462684687" Jul 06 21:16:01.816332 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:01.816285 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:16:07.895501 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:07.895448 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:16:07.895899 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:07.895528 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:16:07.900196 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:07.900173 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:16:08.725412 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:08.725384 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:16:08.771431 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:08.771396 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-559b687498-trz57"] Jul 06 21:16:22.568638 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:22.568579 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7449c7544b-vgfgs" podUID="bb8c24c7-b20e-4548-a9a8-92d6d0d83004" containerName="console" containerID="cri-o://fb54d2b87661b86deb795f97137413c458e03f969b7e80ee1fbdffee5837a36b" gracePeriod=15 Jul 06 21:16:22.773774 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:22.773741 2581 generic.go:356] "Generic (PLEG): container finished" podID="bb8c24c7-b20e-4548-a9a8-92d6d0d83004" containerID="fb54d2b87661b86deb795f97137413c458e03f969b7e80ee1fbdffee5837a36b" exitCode=0 Jul 06 21:16:22.773774 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:22.773763 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7449c7544b-vgfgs" event={"ID":"bb8c24c7-b20e-4548-a9a8-92d6d0d83004","Type":"ContainerDied","Data":"fb54d2b87661b86deb795f97137413c458e03f969b7e80ee1fbdffee5837a36b"} Jul 06 21:16:22.806161 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:22.806136 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:16:22.942983 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:22.942958 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-oauth-serving-cert\") pod \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " Jul 06 21:16:22.943160 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:22.943003 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-console-oauth-config\") pod \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " Jul 06 21:16:22.943160 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:22.943024 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-574br\" (UniqueName: \"kubernetes.io/projected/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-kube-api-access-574br\") pod \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " Jul 06 21:16:22.943160 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:22.943042 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-console-serving-cert\") pod \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " Jul 06 21:16:22.943160 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:22.943070 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-console-config\") pod \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " Jul 06 21:16:22.943160 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:22.943111 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-trusted-ca-bundle\") pod \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " Jul 06 21:16:22.943160 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:22.943154 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-service-ca\") pod \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\" (UID: \"bb8c24c7-b20e-4548-a9a8-92d6d0d83004\") " Jul 06 21:16:22.943541 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:22.943512 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bb8c24c7-b20e-4548-a9a8-92d6d0d83004" (UID: "bb8c24c7-b20e-4548-a9a8-92d6d0d83004"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:16:22.943654 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:22.943619 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-console-config" (OuterVolumeSpecName: "console-config") pod "bb8c24c7-b20e-4548-a9a8-92d6d0d83004" (UID: "bb8c24c7-b20e-4548-a9a8-92d6d0d83004"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:16:22.943654 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:22.943634 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-service-ca" (OuterVolumeSpecName: "service-ca") pod "bb8c24c7-b20e-4548-a9a8-92d6d0d83004" (UID: "bb8c24c7-b20e-4548-a9a8-92d6d0d83004"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:16:22.943654 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:22.943625 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bb8c24c7-b20e-4548-a9a8-92d6d0d83004" (UID: "bb8c24c7-b20e-4548-a9a8-92d6d0d83004"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:16:22.945247 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:22.945214 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bb8c24c7-b20e-4548-a9a8-92d6d0d83004" (UID: "bb8c24c7-b20e-4548-a9a8-92d6d0d83004"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:16:22.945329 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:22.945236 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bb8c24c7-b20e-4548-a9a8-92d6d0d83004" (UID: "bb8c24c7-b20e-4548-a9a8-92d6d0d83004"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:16:22.945329 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:22.945297 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-kube-api-access-574br" (OuterVolumeSpecName: "kube-api-access-574br") pod "bb8c24c7-b20e-4548-a9a8-92d6d0d83004" (UID: "bb8c24c7-b20e-4548-a9a8-92d6d0d83004"). InnerVolumeSpecName "kube-api-access-574br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:16:23.043781 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:23.043736 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-oauth-serving-cert\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:16:23.043781 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:23.043778 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-console-oauth-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:16:23.043781 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:23.043788 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-574br\" (UniqueName: \"kubernetes.io/projected/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-kube-api-access-574br\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:16:23.043781 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:23.043799 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-console-serving-cert\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:16:23.044031 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:23.043809 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-console-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:16:23.044031 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:23.043817 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-trusted-ca-bundle\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:16:23.044031 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:23.043826 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8c24c7-b20e-4548-a9a8-92d6d0d83004-service-ca\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:16:23.779502 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:23.779474 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7449c7544b-vgfgs" Jul 06 21:16:23.779883 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:23.779475 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7449c7544b-vgfgs" event={"ID":"bb8c24c7-b20e-4548-a9a8-92d6d0d83004","Type":"ContainerDied","Data":"0114880db9b7b561f485bf55b0f2b6707a6f476cdf6de018cf66b0fba278e216"} Jul 06 21:16:23.779883 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:23.779607 2581 scope.go:117] "RemoveContainer" containerID="fb54d2b87661b86deb795f97137413c458e03f969b7e80ee1fbdffee5837a36b" Jul 06 21:16:23.803818 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:23.803792 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7449c7544b-vgfgs"] Jul 06 21:16:23.808866 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:23.808839 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7449c7544b-vgfgs"] Jul 06 21:16:25.786183 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:25.786140 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb8c24c7-b20e-4548-a9a8-92d6d0d83004" path="/var/lib/kubelet/pods/bb8c24c7-b20e-4548-a9a8-92d6d0d83004/volumes" Jul 06 21:16:33.793697 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:33.793634 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-559b687498-trz57" podUID="9a9fa581-2f55-4b56-ac19-d45ac4ebf58d" containerName="console" containerID="cri-o://c819ab9c206d8f3aae773b6e718244de4fb1e51e1cbcdf446332356c2d84cc86" gracePeriod=15 Jul 06 21:16:34.034314 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.034291 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559b687498-trz57" Jul 06 21:16:34.133761 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.133722 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-oauth-serving-cert\") pod \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " Jul 06 21:16:34.133948 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.133766 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjcbq\" (UniqueName: \"kubernetes.io/projected/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-kube-api-access-qjcbq\") pod \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " Jul 06 21:16:34.133948 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.133802 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-service-ca\") pod \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " Jul 06 21:16:34.133948 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.133830 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-console-serving-cert\") pod \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " Jul 06 21:16:34.133948 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.133864 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-trusted-ca-bundle\") pod \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " Jul 06 21:16:34.133948 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.133887 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-console-oauth-config\") pod \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " Jul 06 21:16:34.134190 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.134066 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-console-config\") pod \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\" (UID: \"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d\") " Jul 06 21:16:34.134392 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.134355 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-service-ca" (OuterVolumeSpecName: "service-ca") pod "9a9fa581-2f55-4b56-ac19-d45ac4ebf58d" (UID: "9a9fa581-2f55-4b56-ac19-d45ac4ebf58d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:16:34.134392 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.134376 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9a9fa581-2f55-4b56-ac19-d45ac4ebf58d" (UID: "9a9fa581-2f55-4b56-ac19-d45ac4ebf58d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:16:34.134561 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.134425 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9a9fa581-2f55-4b56-ac19-d45ac4ebf58d" (UID: "9a9fa581-2f55-4b56-ac19-d45ac4ebf58d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:16:34.134561 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.134442 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-console-config" (OuterVolumeSpecName: "console-config") pod "9a9fa581-2f55-4b56-ac19-d45ac4ebf58d" (UID: "9a9fa581-2f55-4b56-ac19-d45ac4ebf58d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:16:34.136013 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.135978 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9a9fa581-2f55-4b56-ac19-d45ac4ebf58d" (UID: "9a9fa581-2f55-4b56-ac19-d45ac4ebf58d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:16:34.136117 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.136014 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9a9fa581-2f55-4b56-ac19-d45ac4ebf58d" (UID: "9a9fa581-2f55-4b56-ac19-d45ac4ebf58d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:16:34.136117 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.136023 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-kube-api-access-qjcbq" (OuterVolumeSpecName: "kube-api-access-qjcbq") pod "9a9fa581-2f55-4b56-ac19-d45ac4ebf58d" (UID: "9a9fa581-2f55-4b56-ac19-d45ac4ebf58d"). InnerVolumeSpecName "kube-api-access-qjcbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:16:34.235330 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.235298 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-oauth-serving-cert\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:16:34.235330 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.235329 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjcbq\" (UniqueName: \"kubernetes.io/projected/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-kube-api-access-qjcbq\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:16:34.235539 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.235368 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-service-ca\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:16:34.235539 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.235394 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-console-serving-cert\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:16:34.235539 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.235402 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-trusted-ca-bundle\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:16:34.235539 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.235413 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-console-oauth-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:16:34.235539 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.235421 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d-console-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:16:34.822135 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.822105 2581 generic.go:356] "Generic (PLEG): container finished" podID="9a9fa581-2f55-4b56-ac19-d45ac4ebf58d" containerID="c819ab9c206d8f3aae773b6e718244de4fb1e51e1cbcdf446332356c2d84cc86" exitCode=0 Jul 06 21:16:34.822619 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.822145 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559b687498-trz57" Jul 06 21:16:34.822619 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.822195 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559b687498-trz57" event={"ID":"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d","Type":"ContainerDied","Data":"c819ab9c206d8f3aae773b6e718244de4fb1e51e1cbcdf446332356c2d84cc86"} Jul 06 21:16:34.822619 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.822254 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559b687498-trz57" event={"ID":"9a9fa581-2f55-4b56-ac19-d45ac4ebf58d","Type":"ContainerDied","Data":"26b086f0084e27c6a0c0d155b956075edfab4d3fa80bba8808ec357f1c78c237"} Jul 06 21:16:34.822619 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.822286 2581 scope.go:117] "RemoveContainer" containerID="c819ab9c206d8f3aae773b6e718244de4fb1e51e1cbcdf446332356c2d84cc86" Jul 06 21:16:34.830766 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.830751 2581 scope.go:117] "RemoveContainer" containerID="c819ab9c206d8f3aae773b6e718244de4fb1e51e1cbcdf446332356c2d84cc86" Jul 06 21:16:34.831002 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:16:34.830986 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c819ab9c206d8f3aae773b6e718244de4fb1e51e1cbcdf446332356c2d84cc86\": container with ID starting with c819ab9c206d8f3aae773b6e718244de4fb1e51e1cbcdf446332356c2d84cc86 not found: ID does not exist" containerID="c819ab9c206d8f3aae773b6e718244de4fb1e51e1cbcdf446332356c2d84cc86" Jul 06 21:16:34.831067 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.831013 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c819ab9c206d8f3aae773b6e718244de4fb1e51e1cbcdf446332356c2d84cc86"} err="failed to get container status \"c819ab9c206d8f3aae773b6e718244de4fb1e51e1cbcdf446332356c2d84cc86\": rpc error: code = NotFound desc = could not find container \"c819ab9c206d8f3aae773b6e718244de4fb1e51e1cbcdf446332356c2d84cc86\": container with ID starting with c819ab9c206d8f3aae773b6e718244de4fb1e51e1cbcdf446332356c2d84cc86 not found: ID does not exist" Jul 06 21:16:34.842476 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.842456 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-559b687498-trz57"] Jul 06 21:16:34.845824 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:34.845803 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-559b687498-trz57"] Jul 06 21:16:35.786678 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:35.786591 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a9fa581-2f55-4b56-ac19-d45ac4ebf58d" path="/var/lib/kubelet/pods/9a9fa581-2f55-4b56-ac19-d45ac4ebf58d/volumes" Jul 06 21:16:46.311086 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.311051 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm"] Jul 06 21:16:46.311495 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.311355 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb8c24c7-b20e-4548-a9a8-92d6d0d83004" containerName="console" Jul 06 21:16:46.311495 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.311368 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb8c24c7-b20e-4548-a9a8-92d6d0d83004" containerName="console" Jul 06 21:16:46.311495 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.311378 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a9fa581-2f55-4b56-ac19-d45ac4ebf58d" containerName="console" Jul 06 21:16:46.311495 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.311384 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a9fa581-2f55-4b56-ac19-d45ac4ebf58d" containerName="console" Jul 06 21:16:46.311495 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.311441 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a9fa581-2f55-4b56-ac19-d45ac4ebf58d" containerName="console" Jul 06 21:16:46.311495 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.311448 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb8c24c7-b20e-4548-a9a8-92d6d0d83004" containerName="console" Jul 06 21:16:46.315592 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.315573 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" Jul 06 21:16:46.319808 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.319775 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 21:16:46.319808 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.319783 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 21:16:46.319983 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.319820 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 06 21:16:46.319983 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.319932 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 06 21:16:46.319983 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.319953 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 06 21:16:46.319983 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.319975 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 06 21:16:46.320164 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.320052 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 21:16:46.326022 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.326002 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm"] Jul 06 21:16:46.328018 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.328000 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c4db3b-941a-40d3-a1f7-99b111757964-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69bb8597c-xcnnm\" (UID: \"f6c4db3b-941a-40d3-a1f7-99b111757964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" Jul 06 21:16:46.328120 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.328070 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f6c4db3b-941a-40d3-a1f7-99b111757964-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69bb8597c-xcnnm\" (UID: \"f6c4db3b-941a-40d3-a1f7-99b111757964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" Jul 06 21:16:46.328120 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.328101 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f6c4db3b-941a-40d3-a1f7-99b111757964-hub\") pod \"cluster-proxy-proxy-agent-69bb8597c-xcnnm\" (UID: \"f6c4db3b-941a-40d3-a1f7-99b111757964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" Jul 06 21:16:46.328207 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.328130 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmml4\" (UniqueName: \"kubernetes.io/projected/f6c4db3b-941a-40d3-a1f7-99b111757964-kube-api-access-hmml4\") pod \"cluster-proxy-proxy-agent-69bb8597c-xcnnm\" (UID: \"f6c4db3b-941a-40d3-a1f7-99b111757964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" Jul 06 21:16:46.328243 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.328200 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f6c4db3b-941a-40d3-a1f7-99b111757964-ca\") pod \"cluster-proxy-proxy-agent-69bb8597c-xcnnm\" (UID: \"f6c4db3b-941a-40d3-a1f7-99b111757964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" Jul 06 21:16:46.328273 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.328263 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6c4db3b-941a-40d3-a1f7-99b111757964-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69bb8597c-xcnnm\" (UID: \"f6c4db3b-941a-40d3-a1f7-99b111757964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" Jul 06 21:16:46.428834 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.428800 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f6c4db3b-941a-40d3-a1f7-99b111757964-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69bb8597c-xcnnm\" (UID: \"f6c4db3b-941a-40d3-a1f7-99b111757964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" Jul 06 21:16:46.428834 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.428834 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f6c4db3b-941a-40d3-a1f7-99b111757964-hub\") pod \"cluster-proxy-proxy-agent-69bb8597c-xcnnm\" (UID: \"f6c4db3b-941a-40d3-a1f7-99b111757964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" Jul 06 21:16:46.429092 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.428853 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmml4\" (UniqueName: \"kubernetes.io/projected/f6c4db3b-941a-40d3-a1f7-99b111757964-kube-api-access-hmml4\") pod \"cluster-proxy-proxy-agent-69bb8597c-xcnnm\" (UID: \"f6c4db3b-941a-40d3-a1f7-99b111757964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" Jul 06 21:16:46.429092 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.428872 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f6c4db3b-941a-40d3-a1f7-99b111757964-ca\") pod \"cluster-proxy-proxy-agent-69bb8597c-xcnnm\" (UID: \"f6c4db3b-941a-40d3-a1f7-99b111757964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" Jul 06 21:16:46.429092 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.428894 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6c4db3b-941a-40d3-a1f7-99b111757964-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69bb8597c-xcnnm\" (UID: \"f6c4db3b-941a-40d3-a1f7-99b111757964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" Jul 06 21:16:46.429092 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.428992 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c4db3b-941a-40d3-a1f7-99b111757964-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69bb8597c-xcnnm\" (UID: \"f6c4db3b-941a-40d3-a1f7-99b111757964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" Jul 06 21:16:46.429691 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.429667 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c4db3b-941a-40d3-a1f7-99b111757964-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69bb8597c-xcnnm\" (UID: \"f6c4db3b-941a-40d3-a1f7-99b111757964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" Jul 06 21:16:46.431400 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.431374 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f6c4db3b-941a-40d3-a1f7-99b111757964-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69bb8597c-xcnnm\" (UID: \"f6c4db3b-941a-40d3-a1f7-99b111757964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" Jul 06 21:16:46.431499 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.431387 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f6c4db3b-941a-40d3-a1f7-99b111757964-ca\") pod \"cluster-proxy-proxy-agent-69bb8597c-xcnnm\" (UID: \"f6c4db3b-941a-40d3-a1f7-99b111757964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" Jul 06 21:16:46.431914 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.431892 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f6c4db3b-941a-40d3-a1f7-99b111757964-hub\") pod \"cluster-proxy-proxy-agent-69bb8597c-xcnnm\" (UID: \"f6c4db3b-941a-40d3-a1f7-99b111757964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" Jul 06 21:16:46.431961 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.431917 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6c4db3b-941a-40d3-a1f7-99b111757964-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69bb8597c-xcnnm\" (UID: \"f6c4db3b-941a-40d3-a1f7-99b111757964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" Jul 06 21:16:46.438949 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.438922 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmml4\" (UniqueName: \"kubernetes.io/projected/f6c4db3b-941a-40d3-a1f7-99b111757964-kube-api-access-hmml4\") pod \"cluster-proxy-proxy-agent-69bb8597c-xcnnm\" (UID: \"f6c4db3b-941a-40d3-a1f7-99b111757964\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" Jul 06 21:16:46.643950 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.643914 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" Jul 06 21:16:46.772358 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.772308 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm"] Jul 06 21:16:46.775978 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:16:46.775954 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6c4db3b_941a_40d3_a1f7_99b111757964.slice/crio-2df9fc98f061629d45659cfd5b27130064bdd63992a8f730250b7036d5a8d189 WatchSource:0}: Error finding container 2df9fc98f061629d45659cfd5b27130064bdd63992a8f730250b7036d5a8d189: Status 404 returned error can't find the container with id 2df9fc98f061629d45659cfd5b27130064bdd63992a8f730250b7036d5a8d189 Jul 06 21:16:46.870923 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:46.870884 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" event={"ID":"f6c4db3b-941a-40d3-a1f7-99b111757964","Type":"ContainerStarted","Data":"2df9fc98f061629d45659cfd5b27130064bdd63992a8f730250b7036d5a8d189"} Jul 06 21:16:50.890939 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:50.890910 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" event={"ID":"f6c4db3b-941a-40d3-a1f7-99b111757964","Type":"ContainerStarted","Data":"04a7a9bc091e6194f380e65a9a320c1910fe4702b3af6182eaaccb36f0491344"} Jul 06 21:16:51.897081 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:51.897045 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" event={"ID":"f6c4db3b-941a-40d3-a1f7-99b111757964","Type":"ContainerStarted","Data":"e246f789a0f455d14267b4c9e72ec14abb977d5805f46aa622d8b91dcef20972"} Jul 06 21:16:51.897560 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:51.897095 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" event={"ID":"f6c4db3b-941a-40d3-a1f7-99b111757964","Type":"ContainerStarted","Data":"f9922c437ead12c7b5d7032f9f9c238e5af00bb09d58afa1ec3f9628abff435a"} Jul 06 21:16:51.917634 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:16:51.917584 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69bb8597c-xcnnm" podStartSLOduration=1.8890772679999999 podStartE2EDuration="5.917570279s" totalImagesPullingTime="4.028493011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:16:51.915614933 +0000 UTC m=+236.678527930" watchObservedRunningTime="2026-07-06 21:16:51.917570279 +0000 UTC m=+236.680483283" Jul 06 21:17:55.653778 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:17:55.653747 2581 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 21:18:33.167693 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:18:33.167661 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-xfxpm"] Jul 06 21:18:33.169697 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:18:33.169682 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-xfxpm" Jul 06 21:18:33.172830 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:18:33.172810 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-6pfr9\"" Jul 06 21:18:33.173082 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:18:33.173069 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 06 21:18:33.174271 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:18:33.174252 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Jul 06 21:18:33.174323 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:18:33.174301 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 06 21:18:33.181204 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:18:33.181180 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-xfxpm"] Jul 06 21:18:33.313839 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:18:33.313807 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4a1aa2ef-a835-40cf-81b6-24c6f134fc52-data\") pod \"seaweedfs-64568bcd49-xfxpm\" (UID: \"4a1aa2ef-a835-40cf-81b6-24c6f134fc52\") " pod="kserve/seaweedfs-64568bcd49-xfxpm" Jul 06 21:18:33.314018 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:18:33.313858 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2r7f\" (UniqueName: \"kubernetes.io/projected/4a1aa2ef-a835-40cf-81b6-24c6f134fc52-kube-api-access-w2r7f\") pod \"seaweedfs-64568bcd49-xfxpm\" (UID: \"4a1aa2ef-a835-40cf-81b6-24c6f134fc52\") " pod="kserve/seaweedfs-64568bcd49-xfxpm" Jul 06 21:18:33.415140 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:18:33.415100 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4a1aa2ef-a835-40cf-81b6-24c6f134fc52-data\") pod \"seaweedfs-64568bcd49-xfxpm\" (UID: \"4a1aa2ef-a835-40cf-81b6-24c6f134fc52\") " pod="kserve/seaweedfs-64568bcd49-xfxpm" Jul 06 21:18:33.415321 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:18:33.415161 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2r7f\" (UniqueName: \"kubernetes.io/projected/4a1aa2ef-a835-40cf-81b6-24c6f134fc52-kube-api-access-w2r7f\") pod \"seaweedfs-64568bcd49-xfxpm\" (UID: \"4a1aa2ef-a835-40cf-81b6-24c6f134fc52\") " pod="kserve/seaweedfs-64568bcd49-xfxpm" Jul 06 21:18:33.415523 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:18:33.415502 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4a1aa2ef-a835-40cf-81b6-24c6f134fc52-data\") pod \"seaweedfs-64568bcd49-xfxpm\" (UID: \"4a1aa2ef-a835-40cf-81b6-24c6f134fc52\") " pod="kserve/seaweedfs-64568bcd49-xfxpm" Jul 06 21:18:33.425199 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:18:33.425147 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2r7f\" (UniqueName: \"kubernetes.io/projected/4a1aa2ef-a835-40cf-81b6-24c6f134fc52-kube-api-access-w2r7f\") pod \"seaweedfs-64568bcd49-xfxpm\" (UID: \"4a1aa2ef-a835-40cf-81b6-24c6f134fc52\") " pod="kserve/seaweedfs-64568bcd49-xfxpm" Jul 06 21:18:33.479133 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:18:33.479105 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-xfxpm" Jul 06 21:18:33.613212 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:18:33.613179 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-xfxpm"] Jul 06 21:18:33.616011 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:18:33.615986 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1aa2ef_a835_40cf_81b6_24c6f134fc52.slice/crio-cf568df8a171d66e30a3c4b01d6e2cea80b72128dd81c3303a2dd110ca69d9fe WatchSource:0}: Error finding container cf568df8a171d66e30a3c4b01d6e2cea80b72128dd81c3303a2dd110ca69d9fe: Status 404 returned error can't find the container with id cf568df8a171d66e30a3c4b01d6e2cea80b72128dd81c3303a2dd110ca69d9fe Jul 06 21:18:33.617307 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:18:33.617286 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 21:18:34.285402 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:18:34.285371 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-xfxpm" event={"ID":"4a1aa2ef-a835-40cf-81b6-24c6f134fc52","Type":"ContainerStarted","Data":"cf568df8a171d66e30a3c4b01d6e2cea80b72128dd81c3303a2dd110ca69d9fe"} Jul 06 21:18:38.309252 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:18:38.309214 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-xfxpm" event={"ID":"4a1aa2ef-a835-40cf-81b6-24c6f134fc52","Type":"ContainerStarted","Data":"47550b37d0a4deb7b53bb9fbc77b9f8cd377c0b9bd93c222a8578fa5e1049b1e"} Jul 06 21:18:38.309673 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:18:38.309380 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-xfxpm" Jul 06 21:18:38.329384 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:18:38.329314 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-xfxpm" podStartSLOduration=1.250786975 podStartE2EDuration="5.329297939s" totalImagesPullingTime="4.078510964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:18:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:18:38.327728111 +0000 UTC m=+343.090641117" watchObservedRunningTime="2026-07-06 21:18:38.329297939 +0000 UTC m=+343.092210943" Jul 06 21:18:44.315520 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:18:44.315491 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-xfxpm" Jul 06 21:19:43.105211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:19:43.105174 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-dvpz8"] Jul 06 21:19:43.108365 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:19:43.108332 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-dvpz8" Jul 06 21:19:43.110824 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:19:43.110802 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" Jul 06 21:19:43.110955 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:19:43.110855 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-g8x4w\"" Jul 06 21:19:43.118724 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:19:43.118704 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-dvpz8"] Jul 06 21:19:43.202786 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:19:43.202734 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lwpp\" (UniqueName: \"kubernetes.io/projected/5358db15-8985-486f-9106-5e3e1eac81e7-kube-api-access-6lwpp\") pod \"odh-model-controller-84b54b7f97-dvpz8\" (UID: \"5358db15-8985-486f-9106-5e3e1eac81e7\") " pod="kserve/odh-model-controller-84b54b7f97-dvpz8" Jul 06 21:19:43.202786 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:19:43.202779 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5358db15-8985-486f-9106-5e3e1eac81e7-cert\") pod \"odh-model-controller-84b54b7f97-dvpz8\" (UID: \"5358db15-8985-486f-9106-5e3e1eac81e7\") " pod="kserve/odh-model-controller-84b54b7f97-dvpz8" Jul 06 21:19:43.303518 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:19:43.303486 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lwpp\" (UniqueName: \"kubernetes.io/projected/5358db15-8985-486f-9106-5e3e1eac81e7-kube-api-access-6lwpp\") pod \"odh-model-controller-84b54b7f97-dvpz8\" (UID: \"5358db15-8985-486f-9106-5e3e1eac81e7\") " pod="kserve/odh-model-controller-84b54b7f97-dvpz8" Jul 06 21:19:43.303711 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:19:43.303530 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5358db15-8985-486f-9106-5e3e1eac81e7-cert\") pod \"odh-model-controller-84b54b7f97-dvpz8\" (UID: \"5358db15-8985-486f-9106-5e3e1eac81e7\") " pod="kserve/odh-model-controller-84b54b7f97-dvpz8" Jul 06 21:19:43.303759 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:19:43.303717 2581 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 06 21:19:43.303793 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:19:43.303778 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5358db15-8985-486f-9106-5e3e1eac81e7-cert podName:5358db15-8985-486f-9106-5e3e1eac81e7 nodeName:}" failed. No retries permitted until 2026-07-06 21:19:43.803762954 +0000 UTC m=+408.566675939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5358db15-8985-486f-9106-5e3e1eac81e7-cert") pod "odh-model-controller-84b54b7f97-dvpz8" (UID: "5358db15-8985-486f-9106-5e3e1eac81e7") : secret "odh-model-controller-webhook-cert" not found Jul 06 21:19:43.314224 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:19:43.314203 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lwpp\" (UniqueName: \"kubernetes.io/projected/5358db15-8985-486f-9106-5e3e1eac81e7-kube-api-access-6lwpp\") pod \"odh-model-controller-84b54b7f97-dvpz8\" (UID: \"5358db15-8985-486f-9106-5e3e1eac81e7\") " pod="kserve/odh-model-controller-84b54b7f97-dvpz8" Jul 06 21:19:43.807615 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:19:43.807584 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5358db15-8985-486f-9106-5e3e1eac81e7-cert\") pod \"odh-model-controller-84b54b7f97-dvpz8\" (UID: \"5358db15-8985-486f-9106-5e3e1eac81e7\") " pod="kserve/odh-model-controller-84b54b7f97-dvpz8" Jul 06 21:19:43.809989 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:19:43.809955 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5358db15-8985-486f-9106-5e3e1eac81e7-cert\") pod \"odh-model-controller-84b54b7f97-dvpz8\" (UID: \"5358db15-8985-486f-9106-5e3e1eac81e7\") " pod="kserve/odh-model-controller-84b54b7f97-dvpz8" Jul 06 21:19:44.018201 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:19:44.018171 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-dvpz8" Jul 06 21:19:44.150455 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:19:44.150431 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-dvpz8"] Jul 06 21:19:44.152734 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:19:44.152704 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5358db15_8985_486f_9106_5e3e1eac81e7.slice/crio-a078e21a0a8c1b7f0c30abd3aba11b4513890e2bb45f786c80fd8474482df393 WatchSource:0}: Error finding container a078e21a0a8c1b7f0c30abd3aba11b4513890e2bb45f786c80fd8474482df393: Status 404 returned error can't find the container with id a078e21a0a8c1b7f0c30abd3aba11b4513890e2bb45f786c80fd8474482df393 Jul 06 21:19:44.562896 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:19:44.562857 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-dvpz8" event={"ID":"5358db15-8985-486f-9106-5e3e1eac81e7","Type":"ContainerStarted","Data":"a078e21a0a8c1b7f0c30abd3aba11b4513890e2bb45f786c80fd8474482df393"} Jul 06 21:19:47.576940 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:19:47.576890 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-dvpz8" event={"ID":"5358db15-8985-486f-9106-5e3e1eac81e7","Type":"ContainerStarted","Data":"501b7a1df0045eef6f13ea8cefac01dfaa17d6492058e3c0142099f9ee8737a1"} Jul 06 21:19:47.577378 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:19:47.577061 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-dvpz8" Jul 06 21:19:47.593443 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:19:47.593402 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-dvpz8" podStartSLOduration=1.400409666 podStartE2EDuration="4.593388078s" totalImagesPullingTime="3.192978412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:19:47.592932402 +0000 UTC m=+412.355845417" watchObservedRunningTime="2026-07-06 21:19:47.593388078 +0000 UTC m=+412.356301128" Jul 06 21:19:58.583537 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:19:58.583501 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-dvpz8" Jul 06 21:20:58.367618 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.367586 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-794468d945-9tpj5"] Jul 06 21:20:58.369868 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.369846 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.401840 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.401811 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-794468d945-9tpj5"] Jul 06 21:20:58.433557 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.433521 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c22b3e4-48ec-4632-b8d3-d8725f9a7c08-oauth-serving-cert\") pod \"console-794468d945-9tpj5\" (UID: \"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08\") " pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.433557 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.433560 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c22b3e4-48ec-4632-b8d3-d8725f9a7c08-service-ca\") pod \"console-794468d945-9tpj5\" (UID: \"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08\") " pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.433768 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.433618 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c22b3e4-48ec-4632-b8d3-d8725f9a7c08-trusted-ca-bundle\") pod \"console-794468d945-9tpj5\" (UID: \"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08\") " pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.433768 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.433633 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srldl\" (UniqueName: \"kubernetes.io/projected/0c22b3e4-48ec-4632-b8d3-d8725f9a7c08-kube-api-access-srldl\") pod \"console-794468d945-9tpj5\" (UID: \"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08\") " pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.433768 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.433655 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c22b3e4-48ec-4632-b8d3-d8725f9a7c08-console-serving-cert\") pod \"console-794468d945-9tpj5\" (UID: \"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08\") " pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.433768 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.433738 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c22b3e4-48ec-4632-b8d3-d8725f9a7c08-console-oauth-config\") pod \"console-794468d945-9tpj5\" (UID: \"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08\") " pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.433913 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.433777 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c22b3e4-48ec-4632-b8d3-d8725f9a7c08-console-config\") pod \"console-794468d945-9tpj5\" (UID: \"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08\") " pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.535155 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.535113 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c22b3e4-48ec-4632-b8d3-d8725f9a7c08-oauth-serving-cert\") pod \"console-794468d945-9tpj5\" (UID: \"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08\") " pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.535155 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.535158 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c22b3e4-48ec-4632-b8d3-d8725f9a7c08-service-ca\") pod \"console-794468d945-9tpj5\" (UID: \"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08\") " pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.535432 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.535191 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c22b3e4-48ec-4632-b8d3-d8725f9a7c08-trusted-ca-bundle\") pod \"console-794468d945-9tpj5\" (UID: \"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08\") " pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.535432 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.535206 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srldl\" (UniqueName: \"kubernetes.io/projected/0c22b3e4-48ec-4632-b8d3-d8725f9a7c08-kube-api-access-srldl\") pod \"console-794468d945-9tpj5\" (UID: \"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08\") " pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.535432 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.535231 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c22b3e4-48ec-4632-b8d3-d8725f9a7c08-console-serving-cert\") pod \"console-794468d945-9tpj5\" (UID: \"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08\") " pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.535432 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.535270 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c22b3e4-48ec-4632-b8d3-d8725f9a7c08-console-oauth-config\") pod \"console-794468d945-9tpj5\" (UID: \"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08\") " pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.535432 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.535298 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c22b3e4-48ec-4632-b8d3-d8725f9a7c08-console-config\") pod \"console-794468d945-9tpj5\" (UID: \"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08\") " pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.536023 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.535995 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c22b3e4-48ec-4632-b8d3-d8725f9a7c08-service-ca\") pod \"console-794468d945-9tpj5\" (UID: \"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08\") " pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.536137 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.535995 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c22b3e4-48ec-4632-b8d3-d8725f9a7c08-oauth-serving-cert\") pod \"console-794468d945-9tpj5\" (UID: \"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08\") " pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.536137 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.536049 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c22b3e4-48ec-4632-b8d3-d8725f9a7c08-console-config\") pod \"console-794468d945-9tpj5\" (UID: \"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08\") " pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.536137 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.536108 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c22b3e4-48ec-4632-b8d3-d8725f9a7c08-trusted-ca-bundle\") pod \"console-794468d945-9tpj5\" (UID: \"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08\") " pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.537751 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.537723 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c22b3e4-48ec-4632-b8d3-d8725f9a7c08-console-oauth-config\") pod \"console-794468d945-9tpj5\" (UID: \"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08\") " pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.537910 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.537889 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c22b3e4-48ec-4632-b8d3-d8725f9a7c08-console-serving-cert\") pod \"console-794468d945-9tpj5\" (UID: \"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08\") " pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.548244 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.548224 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srldl\" (UniqueName: \"kubernetes.io/projected/0c22b3e4-48ec-4632-b8d3-d8725f9a7c08-kube-api-access-srldl\") pod \"console-794468d945-9tpj5\" (UID: \"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08\") " pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.678647 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.678550 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:20:58.807464 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.807432 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-794468d945-9tpj5"] Jul 06 21:20:58.810829 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:20:58.810802 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c22b3e4_48ec_4632_b8d3_d8725f9a7c08.slice/crio-b473910afde36ab9639ae9c3e4c35ea363a1de5c32ab9ad502f02fe13df32dac WatchSource:0}: Error finding container b473910afde36ab9639ae9c3e4c35ea363a1de5c32ab9ad502f02fe13df32dac: Status 404 returned error can't find the container with id b473910afde36ab9639ae9c3e4c35ea363a1de5c32ab9ad502f02fe13df32dac Jul 06 21:20:58.850796 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:58.850764 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794468d945-9tpj5" event={"ID":"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08","Type":"ContainerStarted","Data":"b473910afde36ab9639ae9c3e4c35ea363a1de5c32ab9ad502f02fe13df32dac"} Jul 06 21:20:59.855941 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:59.855904 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794468d945-9tpj5" event={"ID":"0c22b3e4-48ec-4632-b8d3-d8725f9a7c08","Type":"ContainerStarted","Data":"0feefe26c1f5718ccf6643b34e9a2ddc7bfbf77daed7f59b29eff906f650fe5a"} Jul 06 21:20:59.872739 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:20:59.872692 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-794468d945-9tpj5" podStartSLOduration=1.872676872 podStartE2EDuration="1.872676872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:20:59.872049594 +0000 UTC m=+484.634962598" watchObservedRunningTime="2026-07-06 21:20:59.872676872 +0000 UTC m=+484.635589876" Jul 06 21:21:08.678917 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:08.678879 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:21:08.679439 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:08.679039 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:21:08.683805 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:08.683780 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:21:08.895562 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:08.895534 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-794468d945-9tpj5" Jul 06 21:21:08.951386 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:08.951295 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66b65fc796-l2drm"] Jul 06 21:21:31.597247 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:31.597216 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c"] Jul 06 21:21:31.600375 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:31.600332 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" Jul 06 21:21:31.603647 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:31.603613 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 06 21:21:31.603647 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:31.603634 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2qpmz\"" Jul 06 21:21:31.603833 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:31.603635 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-dec69-kube-rbac-proxy-sar-config\"" Jul 06 21:21:31.603833 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:31.603711 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 06 21:21:31.603912 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:31.603878 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-dec69-predictor-serving-cert\"" Jul 06 21:21:31.611192 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:31.611169 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c"] Jul 06 21:21:31.715248 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:31.715206 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a71b54c-2db6-479c-abfd-4c9c399fa1d5-proxy-tls\") pod \"success-200-isvc-dec69-predictor-6b576ddc58-dnf7c\" (UID: \"8a71b54c-2db6-479c-abfd-4c9c399fa1d5\") " pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" Jul 06 21:21:31.715248 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:31.715250 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a71b54c-2db6-479c-abfd-4c9c399fa1d5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-dec69-predictor-6b576ddc58-dnf7c\" (UID: \"8a71b54c-2db6-479c-abfd-4c9c399fa1d5\") " pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" Jul 06 21:21:31.715479 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:31.715283 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft9z2\" (UniqueName: \"kubernetes.io/projected/8a71b54c-2db6-479c-abfd-4c9c399fa1d5-kube-api-access-ft9z2\") pod \"success-200-isvc-dec69-predictor-6b576ddc58-dnf7c\" (UID: \"8a71b54c-2db6-479c-abfd-4c9c399fa1d5\") " pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" Jul 06 21:21:31.816468 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:31.816426 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a71b54c-2db6-479c-abfd-4c9c399fa1d5-proxy-tls\") pod \"success-200-isvc-dec69-predictor-6b576ddc58-dnf7c\" (UID: \"8a71b54c-2db6-479c-abfd-4c9c399fa1d5\") " pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" Jul 06 21:21:31.816468 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:31.816466 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a71b54c-2db6-479c-abfd-4c9c399fa1d5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-dec69-predictor-6b576ddc58-dnf7c\" (UID: \"8a71b54c-2db6-479c-abfd-4c9c399fa1d5\") " pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" Jul 06 21:21:31.816726 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:31.816589 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft9z2\" (UniqueName: \"kubernetes.io/projected/8a71b54c-2db6-479c-abfd-4c9c399fa1d5-kube-api-access-ft9z2\") pod \"success-200-isvc-dec69-predictor-6b576ddc58-dnf7c\" (UID: \"8a71b54c-2db6-479c-abfd-4c9c399fa1d5\") " pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" Jul 06 21:21:31.817105 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:31.817085 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a71b54c-2db6-479c-abfd-4c9c399fa1d5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-dec69-predictor-6b576ddc58-dnf7c\" (UID: \"8a71b54c-2db6-479c-abfd-4c9c399fa1d5\") " pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" Jul 06 21:21:31.818912 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:31.818889 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a71b54c-2db6-479c-abfd-4c9c399fa1d5-proxy-tls\") pod \"success-200-isvc-dec69-predictor-6b576ddc58-dnf7c\" (UID: \"8a71b54c-2db6-479c-abfd-4c9c399fa1d5\") " pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" Jul 06 21:21:31.832923 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:31.832891 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft9z2\" (UniqueName: \"kubernetes.io/projected/8a71b54c-2db6-479c-abfd-4c9c399fa1d5-kube-api-access-ft9z2\") pod \"success-200-isvc-dec69-predictor-6b576ddc58-dnf7c\" (UID: \"8a71b54c-2db6-479c-abfd-4c9c399fa1d5\") " pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" Jul 06 21:21:31.911112 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:31.911068 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" Jul 06 21:21:32.045581 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:32.045557 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c"] Jul 06 21:21:32.049575 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:21:32.049523 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a71b54c_2db6_479c_abfd_4c9c399fa1d5.slice/crio-8c9de42512056895d7cf922b2c999b7b2cf757065d89a9fd2223124e6e6bc9a0 WatchSource:0}: Error finding container 8c9de42512056895d7cf922b2c999b7b2cf757065d89a9fd2223124e6e6bc9a0: Status 404 returned error can't find the container with id 8c9de42512056895d7cf922b2c999b7b2cf757065d89a9fd2223124e6e6bc9a0 Jul 06 21:21:32.172145 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:32.172065 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f"] Jul 06 21:21:32.175486 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:32.175466 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" Jul 06 21:21:32.178081 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:32.178060 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-dec69-predictor-serving-cert\"" Jul 06 21:21:32.178209 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:32.178191 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-dec69-kube-rbac-proxy-sar-config\"" Jul 06 21:21:32.186602 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:32.186582 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f"] Jul 06 21:21:32.321572 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:32.321541 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/358de845-5ed5-45e7-9e3f-caeb92edc1a7-proxy-tls\") pod \"error-404-isvc-dec69-predictor-799dcc499-rnh4f\" (UID: \"358de845-5ed5-45e7-9e3f-caeb92edc1a7\") " pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" Jul 06 21:21:32.321762 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:32.321587 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/358de845-5ed5-45e7-9e3f-caeb92edc1a7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dec69-predictor-799dcc499-rnh4f\" (UID: \"358de845-5ed5-45e7-9e3f-caeb92edc1a7\") " pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" Jul 06 21:21:32.321762 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:32.321621 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9n47\" (UniqueName: \"kubernetes.io/projected/358de845-5ed5-45e7-9e3f-caeb92edc1a7-kube-api-access-c9n47\") pod \"error-404-isvc-dec69-predictor-799dcc499-rnh4f\" (UID: \"358de845-5ed5-45e7-9e3f-caeb92edc1a7\") " pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" Jul 06 21:21:32.422952 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:32.422870 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/358de845-5ed5-45e7-9e3f-caeb92edc1a7-proxy-tls\") pod \"error-404-isvc-dec69-predictor-799dcc499-rnh4f\" (UID: \"358de845-5ed5-45e7-9e3f-caeb92edc1a7\") " pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" Jul 06 21:21:32.422952 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:32.422921 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/358de845-5ed5-45e7-9e3f-caeb92edc1a7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dec69-predictor-799dcc499-rnh4f\" (UID: \"358de845-5ed5-45e7-9e3f-caeb92edc1a7\") " pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" Jul 06 21:21:32.422952 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:32.422951 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9n47\" (UniqueName: \"kubernetes.io/projected/358de845-5ed5-45e7-9e3f-caeb92edc1a7-kube-api-access-c9n47\") pod \"error-404-isvc-dec69-predictor-799dcc499-rnh4f\" (UID: \"358de845-5ed5-45e7-9e3f-caeb92edc1a7\") " pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" Jul 06 21:21:32.423254 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:21:32.423063 2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-dec69-predictor-serving-cert: secret "error-404-isvc-dec69-predictor-serving-cert" not found Jul 06 21:21:32.423254 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:21:32.423138 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/358de845-5ed5-45e7-9e3f-caeb92edc1a7-proxy-tls podName:358de845-5ed5-45e7-9e3f-caeb92edc1a7 nodeName:}" failed. No retries permitted until 2026-07-06 21:21:32.923118165 +0000 UTC m=+517.686031148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/358de845-5ed5-45e7-9e3f-caeb92edc1a7-proxy-tls") pod "error-404-isvc-dec69-predictor-799dcc499-rnh4f" (UID: "358de845-5ed5-45e7-9e3f-caeb92edc1a7") : secret "error-404-isvc-dec69-predictor-serving-cert" not found Jul 06 21:21:32.423675 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:32.423657 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/358de845-5ed5-45e7-9e3f-caeb92edc1a7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dec69-predictor-799dcc499-rnh4f\" (UID: \"358de845-5ed5-45e7-9e3f-caeb92edc1a7\") " pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" Jul 06 21:21:32.432720 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:32.432693 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9n47\" (UniqueName: \"kubernetes.io/projected/358de845-5ed5-45e7-9e3f-caeb92edc1a7-kube-api-access-c9n47\") pod \"error-404-isvc-dec69-predictor-799dcc499-rnh4f\" (UID: \"358de845-5ed5-45e7-9e3f-caeb92edc1a7\") " pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" Jul 06 21:21:32.931891 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:32.931853 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/358de845-5ed5-45e7-9e3f-caeb92edc1a7-proxy-tls\") pod \"error-404-isvc-dec69-predictor-799dcc499-rnh4f\" (UID: \"358de845-5ed5-45e7-9e3f-caeb92edc1a7\") " pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" Jul 06 21:21:32.936004 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:32.935976 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/358de845-5ed5-45e7-9e3f-caeb92edc1a7-proxy-tls\") pod \"error-404-isvc-dec69-predictor-799dcc499-rnh4f\" (UID: \"358de845-5ed5-45e7-9e3f-caeb92edc1a7\") " pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" Jul 06 21:21:32.998841 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:32.998803 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" event={"ID":"8a71b54c-2db6-479c-abfd-4c9c399fa1d5","Type":"ContainerStarted","Data":"8c9de42512056895d7cf922b2c999b7b2cf757065d89a9fd2223124e6e6bc9a0"} Jul 06 21:21:33.086951 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:33.086683 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" Jul 06 21:21:33.279890 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:33.279794 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f"] Jul 06 21:21:33.974137 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:33.973944 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-66b65fc796-l2drm" podUID="9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e" containerName="console" containerID="cri-o://23f351253e39e3f809cbcfbba85be10a2f6045f975a3563ff38e810bc4bf9e56" gracePeriod=15 Jul 06 21:21:34.020385 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.020259 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" event={"ID":"358de845-5ed5-45e7-9e3f-caeb92edc1a7","Type":"ContainerStarted","Data":"d030dcaf5155132e5c39ba2337b28cca5a7eae1d38584461b6745b79250da3c4"} Jul 06 21:21:34.341903 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.341609 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:21:34.345404 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.344627 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-oauth-serving-cert\") pod \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " Jul 06 21:21:34.345404 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.344680 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-console-config\") pod \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " Jul 06 21:21:34.345404 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.344719 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-trusted-ca-bundle\") pod \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " Jul 06 21:21:34.345404 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.344809 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-service-ca\") pod \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " Jul 06 21:21:34.345404 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.344875 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l56lk\" (UniqueName: \"kubernetes.io/projected/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-kube-api-access-l56lk\") pod \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " Jul 06 21:21:34.345404 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.344910 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-console-serving-cert\") pod \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " Jul 06 21:21:34.345404 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.344942 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-console-oauth-config\") pod \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\" (UID: \"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e\") " Jul 06 21:21:34.347005 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.346930 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e" (UID: "9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:21:34.347130 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.347113 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-service-ca" (OuterVolumeSpecName: "service-ca") pod "9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e" (UID: "9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:21:34.347500 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.347290 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-console-config" (OuterVolumeSpecName: "console-config") pod "9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e" (UID: "9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:21:34.347585 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.347498 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e" (UID: "9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:21:34.357830 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.357796 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e" (UID: "9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:21:34.359242 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.359202 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e" (UID: "9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:21:34.370606 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.370559 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-kube-api-access-l56lk" (OuterVolumeSpecName: "kube-api-access-l56lk") pod "9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e" (UID: "9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e"). InnerVolumeSpecName "kube-api-access-l56lk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:21:34.446157 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.446055 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-console-oauth-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:21:34.446157 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.446082 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-oauth-serving-cert\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:21:34.446157 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.446091 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-console-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:21:34.446157 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.446100 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-trusted-ca-bundle\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:21:34.446157 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.446108 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-service-ca\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:21:34.446157 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.446116 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l56lk\" (UniqueName: \"kubernetes.io/projected/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-kube-api-access-l56lk\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:21:34.446157 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:34.446128 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e-console-serving-cert\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:21:35.035495 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:35.035461 2581 generic.go:356] "Generic (PLEG): container finished" podID="9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e" containerID="23f351253e39e3f809cbcfbba85be10a2f6045f975a3563ff38e810bc4bf9e56" exitCode=0 Jul 06 21:21:35.036216 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:35.035527 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b65fc796-l2drm" event={"ID":"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e","Type":"ContainerDied","Data":"23f351253e39e3f809cbcfbba85be10a2f6045f975a3563ff38e810bc4bf9e56"} Jul 06 21:21:35.036216 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:35.035574 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b65fc796-l2drm" event={"ID":"9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e","Type":"ContainerDied","Data":"2d8d10d9f37092c8e3e07444c14fa31b79f6d9211f6bf4788592ef64d4ab340f"} Jul 06 21:21:35.036216 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:35.035601 2581 scope.go:117] "RemoveContainer" containerID="23f351253e39e3f809cbcfbba85be10a2f6045f975a3563ff38e810bc4bf9e56" Jul 06 21:21:35.036216 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:35.035767 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b65fc796-l2drm" Jul 06 21:21:35.060163 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:35.060016 2581 scope.go:117] "RemoveContainer" containerID="23f351253e39e3f809cbcfbba85be10a2f6045f975a3563ff38e810bc4bf9e56" Jul 06 21:21:35.060665 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:21:35.060597 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23f351253e39e3f809cbcfbba85be10a2f6045f975a3563ff38e810bc4bf9e56\": container with ID starting with 23f351253e39e3f809cbcfbba85be10a2f6045f975a3563ff38e810bc4bf9e56 not found: ID does not exist" containerID="23f351253e39e3f809cbcfbba85be10a2f6045f975a3563ff38e810bc4bf9e56" Jul 06 21:21:35.060665 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:35.060635 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23f351253e39e3f809cbcfbba85be10a2f6045f975a3563ff38e810bc4bf9e56"} err="failed to get container status \"23f351253e39e3f809cbcfbba85be10a2f6045f975a3563ff38e810bc4bf9e56\": rpc error: code = NotFound desc = could not find container \"23f351253e39e3f809cbcfbba85be10a2f6045f975a3563ff38e810bc4bf9e56\": container with ID starting with 23f351253e39e3f809cbcfbba85be10a2f6045f975a3563ff38e810bc4bf9e56 not found: ID does not exist" Jul 06 21:21:35.065085 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:35.064991 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66b65fc796-l2drm"] Jul 06 21:21:35.074840 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:35.074817 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66b65fc796-l2drm"] Jul 06 21:21:35.793256 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:35.793179 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e" path="/var/lib/kubelet/pods/9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e/volumes" Jul 06 21:21:50.128173 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:50.128069 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" event={"ID":"8a71b54c-2db6-479c-abfd-4c9c399fa1d5","Type":"ContainerStarted","Data":"fba7ecb58c3066aacb506b92ebe47afbd130b0269e8d3f74e42cbea1ab49379f"} Jul 06 21:21:50.132675 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:50.132584 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" event={"ID":"358de845-5ed5-45e7-9e3f-caeb92edc1a7","Type":"ContainerStarted","Data":"a3664e9d9fba547ccc42c25eb669c7f2dfc13600fa5c63adc70cb6150e0955db"} Jul 06 21:21:52.143664 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:52.143626 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" event={"ID":"8a71b54c-2db6-479c-abfd-4c9c399fa1d5","Type":"ContainerStarted","Data":"b64b53467d04c7520f2b15b71b3ed6f2309276c91cd8c70f5536386eff9c53c0"} Jul 06 21:21:52.144131 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:52.143791 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" Jul 06 21:21:52.144131 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:52.143818 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" Jul 06 21:21:52.145554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:52.145509 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" podUID="8a71b54c-2db6-479c-abfd-4c9c399fa1d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Jul 06 21:21:52.145554 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:52.145540 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" event={"ID":"358de845-5ed5-45e7-9e3f-caeb92edc1a7","Type":"ContainerStarted","Data":"ff7cb3c7904a7a815660f6f1730e585dc274c88eb3e6a440cda3e7233188bc63"} Jul 06 21:21:52.145764 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:52.145749 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" Jul 06 21:21:52.145801 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:52.145794 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" Jul 06 21:21:52.146723 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:52.146700 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" podUID="358de845-5ed5-45e7-9e3f-caeb92edc1a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Jul 06 21:21:52.163101 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:52.163059 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" podStartSLOduration=1.57489394 podStartE2EDuration="21.163045553s" totalImagesPullingTime="19.588151613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:21:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:21:52.160905923 +0000 UTC m=+536.923818926" watchObservedRunningTime="2026-07-06 21:21:52.163045553 +0000 UTC m=+536.925958556" Jul 06 21:21:52.179559 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:52.179510 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" podStartSLOduration=1.8173003319999999 podStartE2EDuration="20.179492835s" totalImagesPullingTime="18.362192503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:21:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:21:52.177959962 +0000 UTC m=+536.940872980" watchObservedRunningTime="2026-07-06 21:21:52.179492835 +0000 UTC m=+536.942405845" Jul 06 21:21:53.151409 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:53.151361 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" podUID="358de845-5ed5-45e7-9e3f-caeb92edc1a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Jul 06 21:21:53.151854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:53.151482 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" podUID="8a71b54c-2db6-479c-abfd-4c9c399fa1d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Jul 06 21:21:58.156677 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:58.156644 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" Jul 06 21:21:58.157188 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:58.157156 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" podUID="358de845-5ed5-45e7-9e3f-caeb92edc1a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Jul 06 21:21:58.157275 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:58.157217 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" Jul 06 21:21:58.157676 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:21:58.157652 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" podUID="8a71b54c-2db6-479c-abfd-4c9c399fa1d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Jul 06 21:22:08.157635 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:22:08.157571 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" podUID="358de845-5ed5-45e7-9e3f-caeb92edc1a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Jul 06 21:22:08.157635 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:22:08.157599 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" podUID="8a71b54c-2db6-479c-abfd-4c9c399fa1d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Jul 06 21:22:18.157907 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:22:18.157869 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" podUID="8a71b54c-2db6-479c-abfd-4c9c399fa1d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Jul 06 21:22:18.158270 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:22:18.157872 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" podUID="358de845-5ed5-45e7-9e3f-caeb92edc1a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Jul 06 21:22:28.157230 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:22:28.157183 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" podUID="358de845-5ed5-45e7-9e3f-caeb92edc1a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Jul 06 21:22:28.157781 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:22:28.157704 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" podUID="8a71b54c-2db6-479c-abfd-4c9c399fa1d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Jul 06 21:22:38.158090 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:22:38.158052 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" Jul 06 21:22:38.158617 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:22:38.158137 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" Jul 06 21:23:01.902088 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:01.902048 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c"] Jul 06 21:23:01.902612 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:01.902369 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" podUID="8a71b54c-2db6-479c-abfd-4c9c399fa1d5" containerName="kserve-container" containerID="cri-o://fba7ecb58c3066aacb506b92ebe47afbd130b0269e8d3f74e42cbea1ab49379f" gracePeriod=30 Jul 06 21:23:01.903255 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:01.902772 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" podUID="8a71b54c-2db6-479c-abfd-4c9c399fa1d5" containerName="kube-rbac-proxy" containerID="cri-o://b64b53467d04c7520f2b15b71b3ed6f2309276c91cd8c70f5536386eff9c53c0" gracePeriod=30 Jul 06 21:23:01.943855 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:01.943824 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h"] Jul 06 21:23:01.944233 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:01.944216 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e" containerName="console" Jul 06 21:23:01.944309 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:01.944237 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e" containerName="console" Jul 06 21:23:01.944392 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:01.944323 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f0c50e5-e18e-4a6d-b2c5-fb9d624a9b3e" containerName="console" Jul 06 21:23:01.946689 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:01.946671 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" Jul 06 21:23:01.949069 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:01.949044 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0aa94-predictor-serving-cert\"" Jul 06 21:23:01.949181 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:01.949072 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0aa94-kube-rbac-proxy-sar-config\"" Jul 06 21:23:01.956536 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:01.956513 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h"] Jul 06 21:23:02.009746 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.009713 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f"] Jul 06 21:23:02.010046 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.009994 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" podUID="358de845-5ed5-45e7-9e3f-caeb92edc1a7" containerName="kserve-container" containerID="cri-o://a3664e9d9fba547ccc42c25eb669c7f2dfc13600fa5c63adc70cb6150e0955db" gracePeriod=30 Jul 06 21:23:02.010296 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.010012 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" podUID="358de845-5ed5-45e7-9e3f-caeb92edc1a7" containerName="kube-rbac-proxy" containerID="cri-o://ff7cb3c7904a7a815660f6f1730e585dc274c88eb3e6a440cda3e7233188bc63" gracePeriod=30 Jul 06 21:23:02.051589 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.051561 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm"] Jul 06 21:23:02.054634 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.054617 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" Jul 06 21:23:02.056889 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.056869 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0aa94-kube-rbac-proxy-sar-config\"" Jul 06 21:23:02.056970 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.056872 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0aa94-predictor-serving-cert\"" Jul 06 21:23:02.064310 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.064287 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm"] Jul 06 21:23:02.130839 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.130808 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a771b291-e691-457b-8a8e-c63eff33a806-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h\" (UID: \"a771b291-e691-457b-8a8e-c63eff33a806\") " pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" Jul 06 21:23:02.130978 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.130855 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2bk8\" (UniqueName: \"kubernetes.io/projected/a771b291-e691-457b-8a8e-c63eff33a806-kube-api-access-r2bk8\") pod \"success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h\" (UID: \"a771b291-e691-457b-8a8e-c63eff33a806\") " pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" Jul 06 21:23:02.130978 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.130883 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a771b291-e691-457b-8a8e-c63eff33a806-proxy-tls\") pod \"success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h\" (UID: \"a771b291-e691-457b-8a8e-c63eff33a806\") " pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" Jul 06 21:23:02.232195 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.232106 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20a7f7d5-09a6-45e3-8af1-ac5be6efefe4-proxy-tls\") pod \"error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm\" (UID: \"20a7f7d5-09a6-45e3-8af1-ac5be6efefe4\") " pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" Jul 06 21:23:02.232195 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.232154 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a771b291-e691-457b-8a8e-c63eff33a806-proxy-tls\") pod \"success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h\" (UID: \"a771b291-e691-457b-8a8e-c63eff33a806\") " pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" Jul 06 21:23:02.232423 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:23:02.232245 2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-serving-cert: secret "success-200-isvc-0aa94-predictor-serving-cert" not found Jul 06 21:23:02.232423 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.232244 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20a7f7d5-09a6-45e3-8af1-ac5be6efefe4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm\" (UID: \"20a7f7d5-09a6-45e3-8af1-ac5be6efefe4\") " pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" Jul 06 21:23:02.232423 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:23:02.232298 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a771b291-e691-457b-8a8e-c63eff33a806-proxy-tls podName:a771b291-e691-457b-8a8e-c63eff33a806 nodeName:}" failed. No retries permitted until 2026-07-06 21:23:02.732282439 +0000 UTC m=+607.495195421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a771b291-e691-457b-8a8e-c63eff33a806-proxy-tls") pod "success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" (UID: "a771b291-e691-457b-8a8e-c63eff33a806") : secret "success-200-isvc-0aa94-predictor-serving-cert" not found Jul 06 21:23:02.232423 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.232392 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a771b291-e691-457b-8a8e-c63eff33a806-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h\" (UID: \"a771b291-e691-457b-8a8e-c63eff33a806\") " pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" Jul 06 21:23:02.232655 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.232435 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnp9n\" (UniqueName: \"kubernetes.io/projected/20a7f7d5-09a6-45e3-8af1-ac5be6efefe4-kube-api-access-jnp9n\") pod \"error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm\" (UID: \"20a7f7d5-09a6-45e3-8af1-ac5be6efefe4\") " pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" Jul 06 21:23:02.232655 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.232458 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2bk8\" (UniqueName: \"kubernetes.io/projected/a771b291-e691-457b-8a8e-c63eff33a806-kube-api-access-r2bk8\") pod \"success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h\" (UID: \"a771b291-e691-457b-8a8e-c63eff33a806\") " pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" Jul 06 21:23:02.233003 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.232987 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a771b291-e691-457b-8a8e-c63eff33a806-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h\" (UID: \"a771b291-e691-457b-8a8e-c63eff33a806\") " pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" Jul 06 21:23:02.241475 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.241451 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2bk8\" (UniqueName: \"kubernetes.io/projected/a771b291-e691-457b-8a8e-c63eff33a806-kube-api-access-r2bk8\") pod \"success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h\" (UID: \"a771b291-e691-457b-8a8e-c63eff33a806\") " pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" Jul 06 21:23:02.333843 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.333799 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnp9n\" (UniqueName: \"kubernetes.io/projected/20a7f7d5-09a6-45e3-8af1-ac5be6efefe4-kube-api-access-jnp9n\") pod \"error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm\" (UID: \"20a7f7d5-09a6-45e3-8af1-ac5be6efefe4\") " pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" Jul 06 21:23:02.334028 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.333856 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20a7f7d5-09a6-45e3-8af1-ac5be6efefe4-proxy-tls\") pod \"error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm\" (UID: \"20a7f7d5-09a6-45e3-8af1-ac5be6efefe4\") " pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" Jul 06 21:23:02.334028 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.333938 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20a7f7d5-09a6-45e3-8af1-ac5be6efefe4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm\" (UID: \"20a7f7d5-09a6-45e3-8af1-ac5be6efefe4\") " pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" Jul 06 21:23:02.334138 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:23:02.334022 2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-serving-cert: secret "error-404-isvc-0aa94-predictor-serving-cert" not found Jul 06 21:23:02.334138 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:23:02.334093 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20a7f7d5-09a6-45e3-8af1-ac5be6efefe4-proxy-tls podName:20a7f7d5-09a6-45e3-8af1-ac5be6efefe4 nodeName:}" failed. No retries permitted until 2026-07-06 21:23:02.834074451 +0000 UTC m=+607.596987446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/20a7f7d5-09a6-45e3-8af1-ac5be6efefe4-proxy-tls") pod "error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" (UID: "20a7f7d5-09a6-45e3-8af1-ac5be6efefe4") : secret "error-404-isvc-0aa94-predictor-serving-cert" not found Jul 06 21:23:02.334620 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.334601 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20a7f7d5-09a6-45e3-8af1-ac5be6efefe4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm\" (UID: \"20a7f7d5-09a6-45e3-8af1-ac5be6efefe4\") " pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" Jul 06 21:23:02.342153 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.342129 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnp9n\" (UniqueName: \"kubernetes.io/projected/20a7f7d5-09a6-45e3-8af1-ac5be6efefe4-kube-api-access-jnp9n\") pod \"error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm\" (UID: \"20a7f7d5-09a6-45e3-8af1-ac5be6efefe4\") " pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" Jul 06 21:23:02.436881 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.436851 2581 generic.go:356] "Generic (PLEG): container finished" podID="8a71b54c-2db6-479c-abfd-4c9c399fa1d5" containerID="b64b53467d04c7520f2b15b71b3ed6f2309276c91cd8c70f5536386eff9c53c0" exitCode=2 Jul 06 21:23:02.437062 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.436931 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" event={"ID":"8a71b54c-2db6-479c-abfd-4c9c399fa1d5","Type":"ContainerDied","Data":"b64b53467d04c7520f2b15b71b3ed6f2309276c91cd8c70f5536386eff9c53c0"} Jul 06 21:23:02.438691 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.438670 2581 generic.go:356] "Generic (PLEG): container finished" podID="358de845-5ed5-45e7-9e3f-caeb92edc1a7" containerID="ff7cb3c7904a7a815660f6f1730e585dc274c88eb3e6a440cda3e7233188bc63" exitCode=2 Jul 06 21:23:02.438779 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.438735 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" event={"ID":"358de845-5ed5-45e7-9e3f-caeb92edc1a7","Type":"ContainerDied","Data":"ff7cb3c7904a7a815660f6f1730e585dc274c88eb3e6a440cda3e7233188bc63"} Jul 06 21:23:02.737755 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.737717 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a771b291-e691-457b-8a8e-c63eff33a806-proxy-tls\") pod \"success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h\" (UID: \"a771b291-e691-457b-8a8e-c63eff33a806\") " pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" Jul 06 21:23:02.740185 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.740161 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a771b291-e691-457b-8a8e-c63eff33a806-proxy-tls\") pod \"success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h\" (UID: \"a771b291-e691-457b-8a8e-c63eff33a806\") " pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" Jul 06 21:23:02.838818 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.838774 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20a7f7d5-09a6-45e3-8af1-ac5be6efefe4-proxy-tls\") pod \"error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm\" (UID: \"20a7f7d5-09a6-45e3-8af1-ac5be6efefe4\") " pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" Jul 06 21:23:02.841141 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.841115 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20a7f7d5-09a6-45e3-8af1-ac5be6efefe4-proxy-tls\") pod \"error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm\" (UID: \"20a7f7d5-09a6-45e3-8af1-ac5be6efefe4\") " pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" Jul 06 21:23:02.858180 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.858151 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" Jul 06 21:23:02.964865 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.964818 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" Jul 06 21:23:02.989136 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:02.989114 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h"] Jul 06 21:23:02.992156 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:23:02.992132 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda771b291_e691_457b_8a8e_c63eff33a806.slice/crio-407ecb19c9b660643ada9e7f6a7749737ca7b8f3fa8b88d522ac44f352933222 WatchSource:0}: Error finding container 407ecb19c9b660643ada9e7f6a7749737ca7b8f3fa8b88d522ac44f352933222: Status 404 returned error can't find the container with id 407ecb19c9b660643ada9e7f6a7749737ca7b8f3fa8b88d522ac44f352933222 Jul 06 21:23:03.104272 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:03.104250 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm"] Jul 06 21:23:03.106704 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:23:03.106674 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a7f7d5_09a6_45e3_8af1_ac5be6efefe4.slice/crio-32dfab7ea513d924948c57d7f666ce4a821e48fa8f513460f8ab5c4bad594078 WatchSource:0}: Error finding container 32dfab7ea513d924948c57d7f666ce4a821e48fa8f513460f8ab5c4bad594078: Status 404 returned error can't find the container with id 32dfab7ea513d924948c57d7f666ce4a821e48fa8f513460f8ab5c4bad594078 Jul 06 21:23:03.152566 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:03.152422 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" podUID="358de845-5ed5-45e7-9e3f-caeb92edc1a7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused" Jul 06 21:23:03.152566 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:03.152488 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" podUID="8a71b54c-2db6-479c-abfd-4c9c399fa1d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused" Jul 06 21:23:03.445731 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:03.445695 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" event={"ID":"20a7f7d5-09a6-45e3-8af1-ac5be6efefe4","Type":"ContainerStarted","Data":"0f37040bc7eb7ddc9eba87fb11eacdd51d2354046b977cf094a003dc4b37237e"} Jul 06 21:23:03.445942 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:03.445752 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" event={"ID":"20a7f7d5-09a6-45e3-8af1-ac5be6efefe4","Type":"ContainerStarted","Data":"7ea1e61f79cfe63d98792a9328c8b104e6b42f1c9b9bd2b327f05a68768dd2ea"} Jul 06 21:23:03.445942 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:03.445775 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" event={"ID":"20a7f7d5-09a6-45e3-8af1-ac5be6efefe4","Type":"ContainerStarted","Data":"32dfab7ea513d924948c57d7f666ce4a821e48fa8f513460f8ab5c4bad594078"} Jul 06 21:23:03.445942 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:03.445807 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" Jul 06 21:23:03.445942 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:03.445836 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" Jul 06 21:23:03.447475 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:03.447444 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" podUID="20a7f7d5-09a6-45e3-8af1-ac5be6efefe4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Jul 06 21:23:03.449106 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:03.447901 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" event={"ID":"a771b291-e691-457b-8a8e-c63eff33a806","Type":"ContainerStarted","Data":"1ded7ccb84c34c385b5f775b99ddaa89d34f9d1acb7dd08d59acdff1904233c3"} Jul 06 21:23:03.449106 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:03.447968 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" Jul 06 21:23:03.449106 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:03.448001 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" event={"ID":"a771b291-e691-457b-8a8e-c63eff33a806","Type":"ContainerStarted","Data":"58162ea6ce4503034a1cc48097c9e28938bd055315622fe784b30debdbf2d2ef"} Jul 06 21:23:03.449106 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:03.448207 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" Jul 06 21:23:03.449106 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:03.448248 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" event={"ID":"a771b291-e691-457b-8a8e-c63eff33a806","Type":"ContainerStarted","Data":"407ecb19c9b660643ada9e7f6a7749737ca7b8f3fa8b88d522ac44f352933222"} Jul 06 21:23:03.449493 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:03.449131 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" podUID="a771b291-e691-457b-8a8e-c63eff33a806" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Jul 06 21:23:03.463616 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:03.463570 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" podStartSLOduration=1.463556145 podStartE2EDuration="1.463556145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:23:03.462067662 +0000 UTC m=+608.224980666" watchObservedRunningTime="2026-07-06 21:23:03.463556145 +0000 UTC m=+608.226469149" Jul 06 21:23:03.479388 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:03.479332 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" podStartSLOduration=2.479303732 podStartE2EDuration="2.479303732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:23:03.477879656 +0000 UTC m=+608.240792661" watchObservedRunningTime="2026-07-06 21:23:03.479303732 +0000 UTC m=+608.242216732" Jul 06 21:23:04.453364 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:04.453314 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" podUID="a771b291-e691-457b-8a8e-c63eff33a806" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Jul 06 21:23:04.453956 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:04.453403 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" podUID="20a7f7d5-09a6-45e3-8af1-ac5be6efefe4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Jul 06 21:23:05.351766 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.351729 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" Jul 06 21:23:05.361818 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.361792 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ft9z2\" (UniqueName: \"kubernetes.io/projected/8a71b54c-2db6-479c-abfd-4c9c399fa1d5-kube-api-access-ft9z2\") pod \"8a71b54c-2db6-479c-abfd-4c9c399fa1d5\" (UID: \"8a71b54c-2db6-479c-abfd-4c9c399fa1d5\") " Jul 06 21:23:05.361979 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.361829 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a71b54c-2db6-479c-abfd-4c9c399fa1d5-proxy-tls\") pod \"8a71b54c-2db6-479c-abfd-4c9c399fa1d5\" (UID: \"8a71b54c-2db6-479c-abfd-4c9c399fa1d5\") " Jul 06 21:23:05.361979 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.361854 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a71b54c-2db6-479c-abfd-4c9c399fa1d5-kube-rbac-proxy-sar-config\") pod \"8a71b54c-2db6-479c-abfd-4c9c399fa1d5\" (UID: \"8a71b54c-2db6-479c-abfd-4c9c399fa1d5\") " Jul 06 21:23:05.362228 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.362206 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a71b54c-2db6-479c-abfd-4c9c399fa1d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8a71b54c-2db6-479c-abfd-4c9c399fa1d5" (UID: "8a71b54c-2db6-479c-abfd-4c9c399fa1d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:23:05.364036 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.364006 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a71b54c-2db6-479c-abfd-4c9c399fa1d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8a71b54c-2db6-479c-abfd-4c9c399fa1d5" (UID: "8a71b54c-2db6-479c-abfd-4c9c399fa1d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:23:05.364123 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.364061 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a71b54c-2db6-479c-abfd-4c9c399fa1d5-kube-api-access-ft9z2" (OuterVolumeSpecName: "kube-api-access-ft9z2") pod "8a71b54c-2db6-479c-abfd-4c9c399fa1d5" (UID: "8a71b54c-2db6-479c-abfd-4c9c399fa1d5"). InnerVolumeSpecName "kube-api-access-ft9z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:23:05.458967 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.458876 2581 generic.go:356] "Generic (PLEG): container finished" podID="8a71b54c-2db6-479c-abfd-4c9c399fa1d5" containerID="fba7ecb58c3066aacb506b92ebe47afbd130b0269e8d3f74e42cbea1ab49379f" exitCode=0 Jul 06 21:23:05.458967 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.458925 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" Jul 06 21:23:05.458967 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.458951 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" event={"ID":"8a71b54c-2db6-479c-abfd-4c9c399fa1d5","Type":"ContainerDied","Data":"fba7ecb58c3066aacb506b92ebe47afbd130b0269e8d3f74e42cbea1ab49379f"} Jul 06 21:23:05.459492 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.459031 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c" event={"ID":"8a71b54c-2db6-479c-abfd-4c9c399fa1d5","Type":"ContainerDied","Data":"8c9de42512056895d7cf922b2c999b7b2cf757065d89a9fd2223124e6e6bc9a0"} Jul 06 21:23:05.459492 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.459059 2581 scope.go:117] "RemoveContainer" containerID="b64b53467d04c7520f2b15b71b3ed6f2309276c91cd8c70f5536386eff9c53c0" Jul 06 21:23:05.462565 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.462539 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft9z2\" (UniqueName: \"kubernetes.io/projected/8a71b54c-2db6-479c-abfd-4c9c399fa1d5-kube-api-access-ft9z2\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:23:05.462687 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.462569 2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a71b54c-2db6-479c-abfd-4c9c399fa1d5-proxy-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:23:05.462687 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.462585 2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a71b54c-2db6-479c-abfd-4c9c399fa1d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:23:05.467476 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.467456 2581 scope.go:117] "RemoveContainer" containerID="fba7ecb58c3066aacb506b92ebe47afbd130b0269e8d3f74e42cbea1ab49379f" Jul 06 21:23:05.474671 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.474656 2581 scope.go:117] "RemoveContainer" containerID="b64b53467d04c7520f2b15b71b3ed6f2309276c91cd8c70f5536386eff9c53c0" Jul 06 21:23:05.474897 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:23:05.474881 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b64b53467d04c7520f2b15b71b3ed6f2309276c91cd8c70f5536386eff9c53c0\": container with ID starting with b64b53467d04c7520f2b15b71b3ed6f2309276c91cd8c70f5536386eff9c53c0 not found: ID does not exist" containerID="b64b53467d04c7520f2b15b71b3ed6f2309276c91cd8c70f5536386eff9c53c0" Jul 06 21:23:05.474945 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.474904 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b64b53467d04c7520f2b15b71b3ed6f2309276c91cd8c70f5536386eff9c53c0"} err="failed to get container status \"b64b53467d04c7520f2b15b71b3ed6f2309276c91cd8c70f5536386eff9c53c0\": rpc error: code = NotFound desc = could not find container \"b64b53467d04c7520f2b15b71b3ed6f2309276c91cd8c70f5536386eff9c53c0\": container with ID starting with b64b53467d04c7520f2b15b71b3ed6f2309276c91cd8c70f5536386eff9c53c0 not found: ID does not exist" Jul 06 21:23:05.474945 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.474919 2581 scope.go:117] "RemoveContainer" containerID="fba7ecb58c3066aacb506b92ebe47afbd130b0269e8d3f74e42cbea1ab49379f" Jul 06 21:23:05.475159 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:23:05.475141 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fba7ecb58c3066aacb506b92ebe47afbd130b0269e8d3f74e42cbea1ab49379f\": container with ID starting with fba7ecb58c3066aacb506b92ebe47afbd130b0269e8d3f74e42cbea1ab49379f not found: ID does not exist" containerID="fba7ecb58c3066aacb506b92ebe47afbd130b0269e8d3f74e42cbea1ab49379f" Jul 06 21:23:05.475206 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.475164 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fba7ecb58c3066aacb506b92ebe47afbd130b0269e8d3f74e42cbea1ab49379f"} err="failed to get container status \"fba7ecb58c3066aacb506b92ebe47afbd130b0269e8d3f74e42cbea1ab49379f\": rpc error: code = NotFound desc = could not find container \"fba7ecb58c3066aacb506b92ebe47afbd130b0269e8d3f74e42cbea1ab49379f\": container with ID starting with fba7ecb58c3066aacb506b92ebe47afbd130b0269e8d3f74e42cbea1ab49379f not found: ID does not exist" Jul 06 21:23:05.480678 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.480655 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c"] Jul 06 21:23:05.484863 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.484845 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dec69-predictor-6b576ddc58-dnf7c"] Jul 06 21:23:05.752289 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.752261 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" Jul 06 21:23:05.764725 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.764705 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/358de845-5ed5-45e7-9e3f-caeb92edc1a7-proxy-tls\") pod \"358de845-5ed5-45e7-9e3f-caeb92edc1a7\" (UID: \"358de845-5ed5-45e7-9e3f-caeb92edc1a7\") " Jul 06 21:23:05.764844 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.764750 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9n47\" (UniqueName: \"kubernetes.io/projected/358de845-5ed5-45e7-9e3f-caeb92edc1a7-kube-api-access-c9n47\") pod \"358de845-5ed5-45e7-9e3f-caeb92edc1a7\" (UID: \"358de845-5ed5-45e7-9e3f-caeb92edc1a7\") " Jul 06 21:23:05.764844 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.764781 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/358de845-5ed5-45e7-9e3f-caeb92edc1a7-kube-rbac-proxy-sar-config\") pod \"358de845-5ed5-45e7-9e3f-caeb92edc1a7\" (UID: \"358de845-5ed5-45e7-9e3f-caeb92edc1a7\") " Jul 06 21:23:05.765202 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.765180 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358de845-5ed5-45e7-9e3f-caeb92edc1a7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "358de845-5ed5-45e7-9e3f-caeb92edc1a7" (UID: "358de845-5ed5-45e7-9e3f-caeb92edc1a7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:23:05.766636 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.766620 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358de845-5ed5-45e7-9e3f-caeb92edc1a7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "358de845-5ed5-45e7-9e3f-caeb92edc1a7" (UID: "358de845-5ed5-45e7-9e3f-caeb92edc1a7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:23:05.766698 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.766679 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358de845-5ed5-45e7-9e3f-caeb92edc1a7-kube-api-access-c9n47" (OuterVolumeSpecName: "kube-api-access-c9n47") pod "358de845-5ed5-45e7-9e3f-caeb92edc1a7" (UID: "358de845-5ed5-45e7-9e3f-caeb92edc1a7"). InnerVolumeSpecName "kube-api-access-c9n47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:23:05.787424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.787387 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a71b54c-2db6-479c-abfd-4c9c399fa1d5" path="/var/lib/kubelet/pods/8a71b54c-2db6-479c-abfd-4c9c399fa1d5/volumes" Jul 06 21:23:05.865557 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.865526 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9n47\" (UniqueName: \"kubernetes.io/projected/358de845-5ed5-45e7-9e3f-caeb92edc1a7-kube-api-access-c9n47\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:23:05.865557 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.865552 2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/358de845-5ed5-45e7-9e3f-caeb92edc1a7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:23:05.865557 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:05.865562 2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/358de845-5ed5-45e7-9e3f-caeb92edc1a7-proxy-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:23:06.465244 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:06.465164 2581 generic.go:356] "Generic (PLEG): container finished" podID="358de845-5ed5-45e7-9e3f-caeb92edc1a7" containerID="a3664e9d9fba547ccc42c25eb669c7f2dfc13600fa5c63adc70cb6150e0955db" exitCode=0 Jul 06 21:23:06.465244 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:06.465211 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" Jul 06 21:23:06.465244 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:06.465222 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" event={"ID":"358de845-5ed5-45e7-9e3f-caeb92edc1a7","Type":"ContainerDied","Data":"a3664e9d9fba547ccc42c25eb669c7f2dfc13600fa5c63adc70cb6150e0955db"} Jul 06 21:23:06.465740 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:06.465269 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f" event={"ID":"358de845-5ed5-45e7-9e3f-caeb92edc1a7","Type":"ContainerDied","Data":"d030dcaf5155132e5c39ba2337b28cca5a7eae1d38584461b6745b79250da3c4"} Jul 06 21:23:06.465740 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:06.465298 2581 scope.go:117] "RemoveContainer" containerID="ff7cb3c7904a7a815660f6f1730e585dc274c88eb3e6a440cda3e7233188bc63" Jul 06 21:23:06.473074 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:06.473054 2581 scope.go:117] "RemoveContainer" containerID="a3664e9d9fba547ccc42c25eb669c7f2dfc13600fa5c63adc70cb6150e0955db" Jul 06 21:23:06.480561 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:06.480513 2581 scope.go:117] "RemoveContainer" containerID="ff7cb3c7904a7a815660f6f1730e585dc274c88eb3e6a440cda3e7233188bc63" Jul 06 21:23:06.480954 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:23:06.480802 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff7cb3c7904a7a815660f6f1730e585dc274c88eb3e6a440cda3e7233188bc63\": container with ID starting with ff7cb3c7904a7a815660f6f1730e585dc274c88eb3e6a440cda3e7233188bc63 not found: ID does not exist" containerID="ff7cb3c7904a7a815660f6f1730e585dc274c88eb3e6a440cda3e7233188bc63" Jul 06 21:23:06.480954 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:06.480863 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff7cb3c7904a7a815660f6f1730e585dc274c88eb3e6a440cda3e7233188bc63"} err="failed to get container status \"ff7cb3c7904a7a815660f6f1730e585dc274c88eb3e6a440cda3e7233188bc63\": rpc error: code = NotFound desc = could not find container \"ff7cb3c7904a7a815660f6f1730e585dc274c88eb3e6a440cda3e7233188bc63\": container with ID starting with ff7cb3c7904a7a815660f6f1730e585dc274c88eb3e6a440cda3e7233188bc63 not found: ID does not exist" Jul 06 21:23:06.480954 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:06.480889 2581 scope.go:117] "RemoveContainer" containerID="a3664e9d9fba547ccc42c25eb669c7f2dfc13600fa5c63adc70cb6150e0955db" Jul 06 21:23:06.481247 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:23:06.481225 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3664e9d9fba547ccc42c25eb669c7f2dfc13600fa5c63adc70cb6150e0955db\": container with ID starting with a3664e9d9fba547ccc42c25eb669c7f2dfc13600fa5c63adc70cb6150e0955db not found: ID does not exist" containerID="a3664e9d9fba547ccc42c25eb669c7f2dfc13600fa5c63adc70cb6150e0955db" Jul 06 21:23:06.481325 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:06.481256 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3664e9d9fba547ccc42c25eb669c7f2dfc13600fa5c63adc70cb6150e0955db"} err="failed to get container status \"a3664e9d9fba547ccc42c25eb669c7f2dfc13600fa5c63adc70cb6150e0955db\": rpc error: code = NotFound desc = could not find container \"a3664e9d9fba547ccc42c25eb669c7f2dfc13600fa5c63adc70cb6150e0955db\": container with ID starting with a3664e9d9fba547ccc42c25eb669c7f2dfc13600fa5c63adc70cb6150e0955db not found: ID does not exist" Jul 06 21:23:06.482268 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:06.482250 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f"] Jul 06 21:23:06.485814 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:06.485797 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dec69-predictor-799dcc499-rnh4f"] Jul 06 21:23:07.786777 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:07.786743 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="358de845-5ed5-45e7-9e3f-caeb92edc1a7" path="/var/lib/kubelet/pods/358de845-5ed5-45e7-9e3f-caeb92edc1a7/volumes" Jul 06 21:23:09.457641 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:09.457614 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" Jul 06 21:23:09.458075 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:09.458032 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" podUID="20a7f7d5-09a6-45e3-8af1-ac5be6efefe4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Jul 06 21:23:09.459263 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:09.459238 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" Jul 06 21:23:09.459740 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:09.459717 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" podUID="a771b291-e691-457b-8a8e-c63eff33a806" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Jul 06 21:23:19.458329 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:19.458220 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" podUID="20a7f7d5-09a6-45e3-8af1-ac5be6efefe4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Jul 06 21:23:19.460085 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:19.460060 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" podUID="a771b291-e691-457b-8a8e-c63eff33a806" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Jul 06 21:23:29.458993 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:29.458944 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" podUID="20a7f7d5-09a6-45e3-8af1-ac5be6efefe4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Jul 06 21:23:29.460179 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:29.460148 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" podUID="a771b291-e691-457b-8a8e-c63eff33a806" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Jul 06 21:23:39.458544 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:39.458504 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" podUID="20a7f7d5-09a6-45e3-8af1-ac5be6efefe4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Jul 06 21:23:39.460093 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:39.459711 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" podUID="a771b291-e691-457b-8a8e-c63eff33a806" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Jul 06 21:23:49.459365 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:49.459319 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" Jul 06 21:23:49.460370 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:23:49.460350 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" Jul 06 21:24:01.902080 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:01.902046 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh"] Jul 06 21:24:01.902625 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:01.902605 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a71b54c-2db6-479c-abfd-4c9c399fa1d5" containerName="kube-rbac-proxy" Jul 06 21:24:01.902682 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:01.902629 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a71b54c-2db6-479c-abfd-4c9c399fa1d5" containerName="kube-rbac-proxy" Jul 06 21:24:01.902682 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:01.902655 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a71b54c-2db6-479c-abfd-4c9c399fa1d5" containerName="kserve-container" Jul 06 21:24:01.902682 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:01.902667 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a71b54c-2db6-479c-abfd-4c9c399fa1d5" containerName="kserve-container" Jul 06 21:24:01.902785 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:01.902703 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358de845-5ed5-45e7-9e3f-caeb92edc1a7" containerName="kserve-container" Jul 06 21:24:01.902785 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:01.902712 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="358de845-5ed5-45e7-9e3f-caeb92edc1a7" containerName="kserve-container" Jul 06 21:24:01.902785 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:01.902723 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358de845-5ed5-45e7-9e3f-caeb92edc1a7" containerName="kube-rbac-proxy" Jul 06 21:24:01.902785 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:01.902732 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="358de845-5ed5-45e7-9e3f-caeb92edc1a7" containerName="kube-rbac-proxy" Jul 06 21:24:01.903065 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:01.902813 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a71b54c-2db6-479c-abfd-4c9c399fa1d5" containerName="kserve-container" Jul 06 21:24:01.903065 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:01.902829 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="358de845-5ed5-45e7-9e3f-caeb92edc1a7" containerName="kube-rbac-proxy" Jul 06 21:24:01.903065 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:01.902839 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="358de845-5ed5-45e7-9e3f-caeb92edc1a7" containerName="kserve-container" Jul 06 21:24:01.903065 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:01.902848 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a71b54c-2db6-479c-abfd-4c9c399fa1d5" containerName="kube-rbac-proxy" Jul 06 21:24:01.907619 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:01.907599 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" Jul 06 21:24:01.910935 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:01.910905 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-22b2a-predictor-serving-cert\"" Jul 06 21:24:01.911390 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:01.911373 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-22b2a-kube-rbac-proxy-sar-config\"" Jul 06 21:24:01.915652 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:01.915629 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh"] Jul 06 21:24:01.940591 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:01.940566 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ltkr\" (UniqueName: \"kubernetes.io/projected/0b6edf03-b1e4-4689-84bc-4d973357b579-kube-api-access-4ltkr\") pod \"success-200-isvc-22b2a-predictor-66499f9fc4-mklrh\" (UID: \"0b6edf03-b1e4-4689-84bc-4d973357b579\") " pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" Jul 06 21:24:01.940724 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:01.940611 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b6edf03-b1e4-4689-84bc-4d973357b579-proxy-tls\") pod \"success-200-isvc-22b2a-predictor-66499f9fc4-mklrh\" (UID: \"0b6edf03-b1e4-4689-84bc-4d973357b579\") " pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" Jul 06 21:24:01.940724 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:01.940662 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b6edf03-b1e4-4689-84bc-4d973357b579-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-22b2a-predictor-66499f9fc4-mklrh\" (UID: \"0b6edf03-b1e4-4689-84bc-4d973357b579\") " pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" Jul 06 21:24:02.010269 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.010239 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl"] Jul 06 21:24:02.013598 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.013577 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" Jul 06 21:24:02.015809 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.015788 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-22b2a-kube-rbac-proxy-sar-config\"" Jul 06 21:24:02.015907 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.015833 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-22b2a-predictor-serving-cert\"" Jul 06 21:24:02.022957 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.022933 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl"] Jul 06 21:24:02.041036 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.041003 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ltkr\" (UniqueName: \"kubernetes.io/projected/0b6edf03-b1e4-4689-84bc-4d973357b579-kube-api-access-4ltkr\") pod \"success-200-isvc-22b2a-predictor-66499f9fc4-mklrh\" (UID: \"0b6edf03-b1e4-4689-84bc-4d973357b579\") " pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" Jul 06 21:24:02.041188 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.041051 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrdd7\" (UniqueName: \"kubernetes.io/projected/cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759-kube-api-access-lrdd7\") pod \"error-404-isvc-22b2a-predictor-7b88574587-dq2jl\" (UID: \"cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759\") " pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" Jul 06 21:24:02.041188 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.041094 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-22b2a-predictor-7b88574587-dq2jl\" (UID: \"cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759\") " pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" Jul 06 21:24:02.041188 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.041157 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b6edf03-b1e4-4689-84bc-4d973357b579-proxy-tls\") pod \"success-200-isvc-22b2a-predictor-66499f9fc4-mklrh\" (UID: \"0b6edf03-b1e4-4689-84bc-4d973357b579\") " pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" Jul 06 21:24:02.041321 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.041194 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759-proxy-tls\") pod \"error-404-isvc-22b2a-predictor-7b88574587-dq2jl\" (UID: \"cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759\") " pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" Jul 06 21:24:02.041321 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.041221 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b6edf03-b1e4-4689-84bc-4d973357b579-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-22b2a-predictor-66499f9fc4-mklrh\" (UID: \"0b6edf03-b1e4-4689-84bc-4d973357b579\") " pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" Jul 06 21:24:02.041321 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:24:02.041266 2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-serving-cert: secret "success-200-isvc-22b2a-predictor-serving-cert" not found Jul 06 21:24:02.041501 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:24:02.041324 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6edf03-b1e4-4689-84bc-4d973357b579-proxy-tls podName:0b6edf03-b1e4-4689-84bc-4d973357b579 nodeName:}" failed. No retries permitted until 2026-07-06 21:24:02.541309011 +0000 UTC m=+667.304221995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0b6edf03-b1e4-4689-84bc-4d973357b579-proxy-tls") pod "success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" (UID: "0b6edf03-b1e4-4689-84bc-4d973357b579") : secret "success-200-isvc-22b2a-predictor-serving-cert" not found Jul 06 21:24:02.041891 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.041868 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b6edf03-b1e4-4689-84bc-4d973357b579-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-22b2a-predictor-66499f9fc4-mklrh\" (UID: \"0b6edf03-b1e4-4689-84bc-4d973357b579\") " pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" Jul 06 21:24:02.049203 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.049181 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ltkr\" (UniqueName: \"kubernetes.io/projected/0b6edf03-b1e4-4689-84bc-4d973357b579-kube-api-access-4ltkr\") pod \"success-200-isvc-22b2a-predictor-66499f9fc4-mklrh\" (UID: \"0b6edf03-b1e4-4689-84bc-4d973357b579\") " pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" Jul 06 21:24:02.142486 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.142444 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrdd7\" (UniqueName: \"kubernetes.io/projected/cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759-kube-api-access-lrdd7\") pod \"error-404-isvc-22b2a-predictor-7b88574587-dq2jl\" (UID: \"cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759\") " pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" Jul 06 21:24:02.142486 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.142493 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-22b2a-predictor-7b88574587-dq2jl\" (UID: \"cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759\") " pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" Jul 06 21:24:02.142751 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.142562 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759-proxy-tls\") pod \"error-404-isvc-22b2a-predictor-7b88574587-dq2jl\" (UID: \"cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759\") " pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" Jul 06 21:24:02.143165 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.143141 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-22b2a-predictor-7b88574587-dq2jl\" (UID: \"cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759\") " pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" Jul 06 21:24:02.144982 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.144959 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759-proxy-tls\") pod \"error-404-isvc-22b2a-predictor-7b88574587-dq2jl\" (UID: \"cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759\") " pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" Jul 06 21:24:02.150328 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.150305 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrdd7\" (UniqueName: \"kubernetes.io/projected/cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759-kube-api-access-lrdd7\") pod \"error-404-isvc-22b2a-predictor-7b88574587-dq2jl\" (UID: \"cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759\") " pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" Jul 06 21:24:02.324371 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.324253 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" Jul 06 21:24:02.455571 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.455546 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl"] Jul 06 21:24:02.457722 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:24:02.457694 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbbf9c58_3bf6_4d8f_9367_5cb1b4ba7759.slice/crio-0db5581e30488b98cf0056c1aa88cfce04a8f1f286a589a683f1dec7c0602855 WatchSource:0}: Error finding container 0db5581e30488b98cf0056c1aa88cfce04a8f1f286a589a683f1dec7c0602855: Status 404 returned error can't find the container with id 0db5581e30488b98cf0056c1aa88cfce04a8f1f286a589a683f1dec7c0602855 Jul 06 21:24:02.546658 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.546632 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b6edf03-b1e4-4689-84bc-4d973357b579-proxy-tls\") pod \"success-200-isvc-22b2a-predictor-66499f9fc4-mklrh\" (UID: \"0b6edf03-b1e4-4689-84bc-4d973357b579\") " pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" Jul 06 21:24:02.548882 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.548861 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b6edf03-b1e4-4689-84bc-4d973357b579-proxy-tls\") pod \"success-200-isvc-22b2a-predictor-66499f9fc4-mklrh\" (UID: \"0b6edf03-b1e4-4689-84bc-4d973357b579\") " pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" Jul 06 21:24:02.697151 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.697110 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" event={"ID":"cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759","Type":"ContainerStarted","Data":"608b2bbbd85ef911703e221615c42521476f724f9ebb7192396d09b6a654155b"} Jul 06 21:24:02.697326 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.697183 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" event={"ID":"cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759","Type":"ContainerStarted","Data":"9585a896e2b56dd50326f0230d89d2f62495aa20f2c9854e2a2d073bc389ea25"} Jul 06 21:24:02.697326 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.697217 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" event={"ID":"cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759","Type":"ContainerStarted","Data":"0db5581e30488b98cf0056c1aa88cfce04a8f1f286a589a683f1dec7c0602855"} Jul 06 21:24:02.697435 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.697362 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" Jul 06 21:24:02.716395 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.716327 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" podStartSLOduration=1.716311298 podStartE2EDuration="1.716311298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:24:02.715086864 +0000 UTC m=+667.477999867" watchObservedRunningTime="2026-07-06 21:24:02.716311298 +0000 UTC m=+667.479224301" Jul 06 21:24:02.821036 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.821001 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" Jul 06 21:24:02.965643 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:02.965614 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh"] Jul 06 21:24:02.967134 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:24:02.967103 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6edf03_b1e4_4689_84bc_4d973357b579.slice/crio-15b61d63538ceebd549758c8634e907677b4773da6705c3014cb2027ace0b94a WatchSource:0}: Error finding container 15b61d63538ceebd549758c8634e907677b4773da6705c3014cb2027ace0b94a: Status 404 returned error can't find the container with id 15b61d63538ceebd549758c8634e907677b4773da6705c3014cb2027ace0b94a Jul 06 21:24:03.703272 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:03.703232 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" event={"ID":"0b6edf03-b1e4-4689-84bc-4d973357b579","Type":"ContainerStarted","Data":"51b6d9545436d67a6a9335193c6ec16379a3bc835daaa62a18f8f7d01e8c3d8e"} Jul 06 21:24:03.703507 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:03.703300 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" event={"ID":"0b6edf03-b1e4-4689-84bc-4d973357b579","Type":"ContainerStarted","Data":"7aff816040954ef637c49f45dd1236d32baa0e845c2ec1d7c24c9d4599f9e09a"} Jul 06 21:24:03.703507 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:03.703329 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" event={"ID":"0b6edf03-b1e4-4689-84bc-4d973357b579","Type":"ContainerStarted","Data":"15b61d63538ceebd549758c8634e907677b4773da6705c3014cb2027ace0b94a"} Jul 06 21:24:03.703628 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:03.703518 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" Jul 06 21:24:03.704915 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:03.704888 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" podUID="cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Jul 06 21:24:03.721140 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:03.721098 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" podStartSLOduration=2.721074804 podStartE2EDuration="2.721074804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:24:03.719832485 +0000 UTC m=+668.482745491" watchObservedRunningTime="2026-07-06 21:24:03.721074804 +0000 UTC m=+668.483987808" Jul 06 21:24:04.707895 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:04.707855 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" Jul 06 21:24:04.708373 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:04.707942 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" podUID="cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Jul 06 21:24:04.708373 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:04.708069 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" Jul 06 21:24:04.709404 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:04.709378 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" podUID="0b6edf03-b1e4-4689-84bc-4d973357b579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Jul 06 21:24:05.711890 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:05.711854 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" podUID="0b6edf03-b1e4-4689-84bc-4d973357b579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Jul 06 21:24:09.712748 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:09.712717 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" Jul 06 21:24:09.713198 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:09.713098 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" podUID="cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Jul 06 21:24:10.716730 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:10.716700 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" Jul 06 21:24:10.717183 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:10.717150 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" podUID="0b6edf03-b1e4-4689-84bc-4d973357b579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Jul 06 21:24:19.713605 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:19.713559 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" podUID="cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Jul 06 21:24:20.718012 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:20.717971 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" podUID="0b6edf03-b1e4-4689-84bc-4d973357b579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Jul 06 21:24:29.713941 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:29.713894 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" podUID="cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Jul 06 21:24:30.717327 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:30.717287 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" podUID="0b6edf03-b1e4-4689-84bc-4d973357b579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Jul 06 21:24:39.713540 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:39.713497 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" podUID="cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Jul 06 21:24:40.718102 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:40.718064 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" podUID="0b6edf03-b1e4-4689-84bc-4d973357b579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Jul 06 21:24:49.714360 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:49.714308 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" Jul 06 21:24:50.718175 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:24:50.718143 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" Jul 06 21:32:16.916243 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:16.916213 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch"] Jul 06 21:32:16.919568 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:16.919549 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h"] Jul 06 21:32:16.919675 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:16.919655 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" Jul 06 21:32:16.919741 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:16.919722 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" podUID="a771b291-e691-457b-8a8e-c63eff33a806" containerName="kserve-container" containerID="cri-o://58162ea6ce4503034a1cc48097c9e28938bd055315622fe784b30debdbf2d2ef" gracePeriod=30 Jul 06 21:32:16.919857 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:16.919805 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" podUID="a771b291-e691-457b-8a8e-c63eff33a806" containerName="kube-rbac-proxy" containerID="cri-o://1ded7ccb84c34c385b5f775b99ddaa89d34f9d1acb7dd08d59acdff1904233c3" gracePeriod=30 Jul 06 21:32:16.921996 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:16.921973 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-69e1e-predictor-serving-cert\"" Jul 06 21:32:16.922088 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:16.922005 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-69e1e-kube-rbac-proxy-sar-config\"" Jul 06 21:32:16.930794 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:16.930762 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch"] Jul 06 21:32:16.984018 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:16.983986 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-69e1e-predictor-7568595c68-8t2ch\" (UID: \"fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc\") " pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" Jul 06 21:32:16.984248 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:16.984035 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjp2g\" (UniqueName: \"kubernetes.io/projected/fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc-kube-api-access-xjp2g\") pod \"success-200-isvc-69e1e-predictor-7568595c68-8t2ch\" (UID: \"fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc\") " pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" Jul 06 21:32:16.984248 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:16.984120 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc-proxy-tls\") pod \"success-200-isvc-69e1e-predictor-7568595c68-8t2ch\" (UID: \"fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc\") " pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" Jul 06 21:32:16.989382 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:16.989332 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg"] Jul 06 21:32:16.992983 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:16.992964 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" Jul 06 21:32:16.995402 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:16.995382 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-69e1e-predictor-serving-cert\"" Jul 06 21:32:16.995505 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:16.995410 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-69e1e-kube-rbac-proxy-sar-config\"" Jul 06 21:32:17.003151 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.003125 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg"] Jul 06 21:32:17.063954 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.063919 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm"] Jul 06 21:32:17.064271 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.064186 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" podUID="20a7f7d5-09a6-45e3-8af1-ac5be6efefe4" containerName="kserve-container" containerID="cri-o://7ea1e61f79cfe63d98792a9328c8b104e6b42f1c9b9bd2b327f05a68768dd2ea" gracePeriod=30 Jul 06 21:32:17.064501 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.064413 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" podUID="20a7f7d5-09a6-45e3-8af1-ac5be6efefe4" containerName="kube-rbac-proxy" containerID="cri-o://0f37040bc7eb7ddc9eba87fb11eacdd51d2354046b977cf094a003dc4b37237e" gracePeriod=30 Jul 06 21:32:17.085082 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.085053 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/412725d6-83f6-4641-9885-765577a11eaf-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-69e1e-predictor-68f588fff7-bq6qg\" (UID: \"412725d6-83f6-4641-9885-765577a11eaf\") " pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" Jul 06 21:32:17.085213 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.085114 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-69e1e-predictor-7568595c68-8t2ch\" (UID: \"fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc\") " pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" Jul 06 21:32:17.085213 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.085136 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/412725d6-83f6-4641-9885-765577a11eaf-proxy-tls\") pod \"error-404-isvc-69e1e-predictor-68f588fff7-bq6qg\" (UID: \"412725d6-83f6-4641-9885-765577a11eaf\") " pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" Jul 06 21:32:17.085213 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.085159 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjp2g\" (UniqueName: \"kubernetes.io/projected/fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc-kube-api-access-xjp2g\") pod \"success-200-isvc-69e1e-predictor-7568595c68-8t2ch\" (UID: \"fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc\") " pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" Jul 06 21:32:17.085213 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.085194 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc-proxy-tls\") pod \"success-200-isvc-69e1e-predictor-7568595c68-8t2ch\" (UID: \"fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc\") " pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" Jul 06 21:32:17.085459 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.085231 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrmsg\" (UniqueName: \"kubernetes.io/projected/412725d6-83f6-4641-9885-765577a11eaf-kube-api-access-zrmsg\") pod \"error-404-isvc-69e1e-predictor-68f588fff7-bq6qg\" (UID: \"412725d6-83f6-4641-9885-765577a11eaf\") " pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" Jul 06 21:32:17.085813 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.085793 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-69e1e-predictor-7568595c68-8t2ch\" (UID: \"fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc\") " pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" Jul 06 21:32:17.087657 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.087634 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc-proxy-tls\") pod \"success-200-isvc-69e1e-predictor-7568595c68-8t2ch\" (UID: \"fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc\") " pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" Jul 06 21:32:17.094611 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.094585 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjp2g\" (UniqueName: \"kubernetes.io/projected/fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc-kube-api-access-xjp2g\") pod \"success-200-isvc-69e1e-predictor-7568595c68-8t2ch\" (UID: \"fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc\") " pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" Jul 06 21:32:17.186193 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.186164 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrmsg\" (UniqueName: \"kubernetes.io/projected/412725d6-83f6-4641-9885-765577a11eaf-kube-api-access-zrmsg\") pod \"error-404-isvc-69e1e-predictor-68f588fff7-bq6qg\" (UID: \"412725d6-83f6-4641-9885-765577a11eaf\") " pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" Jul 06 21:32:17.186299 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.186209 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/412725d6-83f6-4641-9885-765577a11eaf-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-69e1e-predictor-68f588fff7-bq6qg\" (UID: \"412725d6-83f6-4641-9885-765577a11eaf\") " pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" Jul 06 21:32:17.186458 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.186427 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/412725d6-83f6-4641-9885-765577a11eaf-proxy-tls\") pod \"error-404-isvc-69e1e-predictor-68f588fff7-bq6qg\" (UID: \"412725d6-83f6-4641-9885-765577a11eaf\") " pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" Jul 06 21:32:17.186808 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.186788 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/412725d6-83f6-4641-9885-765577a11eaf-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-69e1e-predictor-68f588fff7-bq6qg\" (UID: \"412725d6-83f6-4641-9885-765577a11eaf\") " pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" Jul 06 21:32:17.188804 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.188778 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/412725d6-83f6-4641-9885-765577a11eaf-proxy-tls\") pod \"error-404-isvc-69e1e-predictor-68f588fff7-bq6qg\" (UID: \"412725d6-83f6-4641-9885-765577a11eaf\") " pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" Jul 06 21:32:17.198614 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.198589 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrmsg\" (UniqueName: \"kubernetes.io/projected/412725d6-83f6-4641-9885-765577a11eaf-kube-api-access-zrmsg\") pod \"error-404-isvc-69e1e-predictor-68f588fff7-bq6qg\" (UID: \"412725d6-83f6-4641-9885-765577a11eaf\") " pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" Jul 06 21:32:17.240555 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.240526 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" Jul 06 21:32:17.303178 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.303145 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" Jul 06 21:32:17.383708 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.383681 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch"] Jul 06 21:32:17.386143 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:32:17.386117 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdee1a4b_bae2_4f37_8fb9_3acc1f8c64cc.slice/crio-c7550448f13380be5eee8a66a5cc76d3bb6e227f2263c9c3495abb5f1ede44b4 WatchSource:0}: Error finding container c7550448f13380be5eee8a66a5cc76d3bb6e227f2263c9c3495abb5f1ede44b4: Status 404 returned error can't find the container with id c7550448f13380be5eee8a66a5cc76d3bb6e227f2263c9c3495abb5f1ede44b4 Jul 06 21:32:17.448705 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.448681 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg"] Jul 06 21:32:17.451793 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:32:17.451771 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412725d6_83f6_4641_9885_765577a11eaf.slice/crio-493ee55bda3db0357fec38c7571cff44982902f464bd303847ec40ce53bd6871 WatchSource:0}: Error finding container 493ee55bda3db0357fec38c7571cff44982902f464bd303847ec40ce53bd6871: Status 404 returned error can't find the container with id 493ee55bda3db0357fec38c7571cff44982902f464bd303847ec40ce53bd6871 Jul 06 21:32:17.772196 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.772110 2581 generic.go:356] "Generic (PLEG): container finished" podID="20a7f7d5-09a6-45e3-8af1-ac5be6efefe4" containerID="0f37040bc7eb7ddc9eba87fb11eacdd51d2354046b977cf094a003dc4b37237e" exitCode=2 Jul 06 21:32:17.772381 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.772183 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" event={"ID":"20a7f7d5-09a6-45e3-8af1-ac5be6efefe4","Type":"ContainerDied","Data":"0f37040bc7eb7ddc9eba87fb11eacdd51d2354046b977cf094a003dc4b37237e"} Jul 06 21:32:17.773938 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.773917 2581 generic.go:356] "Generic (PLEG): container finished" podID="a771b291-e691-457b-8a8e-c63eff33a806" containerID="1ded7ccb84c34c385b5f775b99ddaa89d34f9d1acb7dd08d59acdff1904233c3" exitCode=2 Jul 06 21:32:17.774036 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.773952 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" event={"ID":"a771b291-e691-457b-8a8e-c63eff33a806","Type":"ContainerDied","Data":"1ded7ccb84c34c385b5f775b99ddaa89d34f9d1acb7dd08d59acdff1904233c3"} Jul 06 21:32:17.775719 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.775687 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" event={"ID":"412725d6-83f6-4641-9885-765577a11eaf","Type":"ContainerStarted","Data":"32b87d25f8feb9e34247c10101714a484ae6fcbb2abf8ffed20a4f5b1f7c0804"} Jul 06 21:32:17.775826 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.775741 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" event={"ID":"412725d6-83f6-4641-9885-765577a11eaf","Type":"ContainerStarted","Data":"7795dc6430bc773a4d8bb995d5cd73ccb0c685cbc50b2e3b8ddd77385df36bab"} Jul 06 21:32:17.775826 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.775772 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" event={"ID":"412725d6-83f6-4641-9885-765577a11eaf","Type":"ContainerStarted","Data":"493ee55bda3db0357fec38c7571cff44982902f464bd303847ec40ce53bd6871"} Jul 06 21:32:17.775934 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.775901 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" Jul 06 21:32:17.777287 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.777267 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" event={"ID":"fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc","Type":"ContainerStarted","Data":"0d805541d7de269b7a61cba6036f70f23eaa551a4e9c98e8a2b9510164e619b8"} Jul 06 21:32:17.777411 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.777316 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" event={"ID":"fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc","Type":"ContainerStarted","Data":"32e3ff29fa06739d765f1dff8f49472a5479a56fd80b69312b24f085a718dc83"} Jul 06 21:32:17.777411 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.777366 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" Jul 06 21:32:17.777411 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.777397 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" event={"ID":"fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc","Type":"ContainerStarted","Data":"c7550448f13380be5eee8a66a5cc76d3bb6e227f2263c9c3495abb5f1ede44b4"} Jul 06 21:32:17.777551 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.777435 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" Jul 06 21:32:17.778833 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.778809 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" podUID="fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Jul 06 21:32:17.796331 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.795997 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" podStartSLOduration=1.7959777890000002 podStartE2EDuration="1.795977789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:32:17.793275794 +0000 UTC m=+1162.556188801" watchObservedRunningTime="2026-07-06 21:32:17.795977789 +0000 UTC m=+1162.558890796" Jul 06 21:32:17.818168 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:17.818114 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" podStartSLOduration=1.8180998339999999 podStartE2EDuration="1.818099834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:32:17.816312082 +0000 UTC m=+1162.579225086" watchObservedRunningTime="2026-07-06 21:32:17.818099834 +0000 UTC m=+1162.581012838" Jul 06 21:32:18.781778 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:18.781746 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" Jul 06 21:32:18.781778 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:18.781747 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" podUID="fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Jul 06 21:32:18.783068 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:18.783039 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" podUID="412725d6-83f6-4641-9885-765577a11eaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Jul 06 21:32:19.453871 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:19.453800 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" podUID="20a7f7d5-09a6-45e3-8af1-ac5be6efefe4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused" Jul 06 21:32:19.454090 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:19.454039 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" podUID="a771b291-e691-457b-8a8e-c63eff33a806" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused" Jul 06 21:32:19.458547 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:19.458513 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" podUID="20a7f7d5-09a6-45e3-8af1-ac5be6efefe4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Jul 06 21:32:19.459695 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:19.459673 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" podUID="a771b291-e691-457b-8a8e-c63eff33a806" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Jul 06 21:32:19.786865 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:19.786780 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" podUID="412725d6-83f6-4641-9885-765577a11eaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Jul 06 21:32:20.700733 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.700674 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" Jul 06 21:32:20.703605 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.703583 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" Jul 06 21:32:20.793509 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.793476 2581 generic.go:356] "Generic (PLEG): container finished" podID="20a7f7d5-09a6-45e3-8af1-ac5be6efefe4" containerID="7ea1e61f79cfe63d98792a9328c8b104e6b42f1c9b9bd2b327f05a68768dd2ea" exitCode=0 Jul 06 21:32:20.793928 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.793528 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" Jul 06 21:32:20.793928 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.793575 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" event={"ID":"20a7f7d5-09a6-45e3-8af1-ac5be6efefe4","Type":"ContainerDied","Data":"7ea1e61f79cfe63d98792a9328c8b104e6b42f1c9b9bd2b327f05a68768dd2ea"} Jul 06 21:32:20.793928 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.793650 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm" event={"ID":"20a7f7d5-09a6-45e3-8af1-ac5be6efefe4","Type":"ContainerDied","Data":"32dfab7ea513d924948c57d7f666ce4a821e48fa8f513460f8ab5c4bad594078"} Jul 06 21:32:20.793928 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.793704 2581 scope.go:117] "RemoveContainer" containerID="0f37040bc7eb7ddc9eba87fb11eacdd51d2354046b977cf094a003dc4b37237e" Jul 06 21:32:20.795222 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.795200 2581 generic.go:356] "Generic (PLEG): container finished" podID="a771b291-e691-457b-8a8e-c63eff33a806" containerID="58162ea6ce4503034a1cc48097c9e28938bd055315622fe784b30debdbf2d2ef" exitCode=0 Jul 06 21:32:20.795325 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.795243 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" Jul 06 21:32:20.795325 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.795291 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" event={"ID":"a771b291-e691-457b-8a8e-c63eff33a806","Type":"ContainerDied","Data":"58162ea6ce4503034a1cc48097c9e28938bd055315622fe784b30debdbf2d2ef"} Jul 06 21:32:20.795439 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.795368 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h" event={"ID":"a771b291-e691-457b-8a8e-c63eff33a806","Type":"ContainerDied","Data":"407ecb19c9b660643ada9e7f6a7749737ca7b8f3fa8b88d522ac44f352933222"} Jul 06 21:32:20.801938 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.801920 2581 scope.go:117] "RemoveContainer" containerID="7ea1e61f79cfe63d98792a9328c8b104e6b42f1c9b9bd2b327f05a68768dd2ea" Jul 06 21:32:20.809400 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.809379 2581 scope.go:117] "RemoveContainer" containerID="0f37040bc7eb7ddc9eba87fb11eacdd51d2354046b977cf094a003dc4b37237e" Jul 06 21:32:20.809709 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:32:20.809680 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f37040bc7eb7ddc9eba87fb11eacdd51d2354046b977cf094a003dc4b37237e\": container with ID starting with 0f37040bc7eb7ddc9eba87fb11eacdd51d2354046b977cf094a003dc4b37237e not found: ID does not exist" containerID="0f37040bc7eb7ddc9eba87fb11eacdd51d2354046b977cf094a003dc4b37237e" Jul 06 21:32:20.809762 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.809719 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f37040bc7eb7ddc9eba87fb11eacdd51d2354046b977cf094a003dc4b37237e"} err="failed to get container status \"0f37040bc7eb7ddc9eba87fb11eacdd51d2354046b977cf094a003dc4b37237e\": rpc error: code = NotFound desc = could not find container \"0f37040bc7eb7ddc9eba87fb11eacdd51d2354046b977cf094a003dc4b37237e\": container with ID starting with 0f37040bc7eb7ddc9eba87fb11eacdd51d2354046b977cf094a003dc4b37237e not found: ID does not exist" Jul 06 21:32:20.809762 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.809737 2581 scope.go:117] "RemoveContainer" containerID="7ea1e61f79cfe63d98792a9328c8b104e6b42f1c9b9bd2b327f05a68768dd2ea" Jul 06 21:32:20.809974 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:32:20.809955 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ea1e61f79cfe63d98792a9328c8b104e6b42f1c9b9bd2b327f05a68768dd2ea\": container with ID starting with 7ea1e61f79cfe63d98792a9328c8b104e6b42f1c9b9bd2b327f05a68768dd2ea not found: ID does not exist" containerID="7ea1e61f79cfe63d98792a9328c8b104e6b42f1c9b9bd2b327f05a68768dd2ea" Jul 06 21:32:20.810022 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.809979 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea1e61f79cfe63d98792a9328c8b104e6b42f1c9b9bd2b327f05a68768dd2ea"} err="failed to get container status \"7ea1e61f79cfe63d98792a9328c8b104e6b42f1c9b9bd2b327f05a68768dd2ea\": rpc error: code = NotFound desc = could not find container \"7ea1e61f79cfe63d98792a9328c8b104e6b42f1c9b9bd2b327f05a68768dd2ea\": container with ID starting with 7ea1e61f79cfe63d98792a9328c8b104e6b42f1c9b9bd2b327f05a68768dd2ea not found: ID does not exist" Jul 06 21:32:20.810022 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.809994 2581 scope.go:117] "RemoveContainer" containerID="1ded7ccb84c34c385b5f775b99ddaa89d34f9d1acb7dd08d59acdff1904233c3" Jul 06 21:32:20.816107 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.816087 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2bk8\" (UniqueName: \"kubernetes.io/projected/a771b291-e691-457b-8a8e-c63eff33a806-kube-api-access-r2bk8\") pod \"a771b291-e691-457b-8a8e-c63eff33a806\" (UID: \"a771b291-e691-457b-8a8e-c63eff33a806\") " Jul 06 21:32:20.816189 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.816145 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a771b291-e691-457b-8a8e-c63eff33a806-proxy-tls\") pod \"a771b291-e691-457b-8a8e-c63eff33a806\" (UID: \"a771b291-e691-457b-8a8e-c63eff33a806\") " Jul 06 21:32:20.816189 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.816175 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20a7f7d5-09a6-45e3-8af1-ac5be6efefe4-kube-rbac-proxy-sar-config\") pod \"20a7f7d5-09a6-45e3-8af1-ac5be6efefe4\" (UID: \"20a7f7d5-09a6-45e3-8af1-ac5be6efefe4\") " Jul 06 21:32:20.816270 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.816207 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnp9n\" (UniqueName: \"kubernetes.io/projected/20a7f7d5-09a6-45e3-8af1-ac5be6efefe4-kube-api-access-jnp9n\") pod \"20a7f7d5-09a6-45e3-8af1-ac5be6efefe4\" (UID: \"20a7f7d5-09a6-45e3-8af1-ac5be6efefe4\") " Jul 06 21:32:20.816270 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.816257 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20a7f7d5-09a6-45e3-8af1-ac5be6efefe4-proxy-tls\") pod \"20a7f7d5-09a6-45e3-8af1-ac5be6efefe4\" (UID: \"20a7f7d5-09a6-45e3-8af1-ac5be6efefe4\") " Jul 06 21:32:20.816368 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.816285 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a771b291-e691-457b-8a8e-c63eff33a806-kube-rbac-proxy-sar-config\") pod \"a771b291-e691-457b-8a8e-c63eff33a806\" (UID: \"a771b291-e691-457b-8a8e-c63eff33a806\") " Jul 06 21:32:20.816729 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.816698 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a771b291-e691-457b-8a8e-c63eff33a806-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a771b291-e691-457b-8a8e-c63eff33a806" (UID: "a771b291-e691-457b-8a8e-c63eff33a806"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:32:20.817051 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.817028 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a7f7d5-09a6-45e3-8af1-ac5be6efefe4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "20a7f7d5-09a6-45e3-8af1-ac5be6efefe4" (UID: "20a7f7d5-09a6-45e3-8af1-ac5be6efefe4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:32:20.817552 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.817530 2581 scope.go:117] "RemoveContainer" containerID="58162ea6ce4503034a1cc48097c9e28938bd055315622fe784b30debdbf2d2ef" Jul 06 21:32:20.818434 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.818398 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a771b291-e691-457b-8a8e-c63eff33a806-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a771b291-e691-457b-8a8e-c63eff33a806" (UID: "a771b291-e691-457b-8a8e-c63eff33a806"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:32:20.818520 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.818463 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a771b291-e691-457b-8a8e-c63eff33a806-kube-api-access-r2bk8" (OuterVolumeSpecName: "kube-api-access-r2bk8") pod "a771b291-e691-457b-8a8e-c63eff33a806" (UID: "a771b291-e691-457b-8a8e-c63eff33a806"). InnerVolumeSpecName "kube-api-access-r2bk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:32:20.818643 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.818627 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a7f7d5-09a6-45e3-8af1-ac5be6efefe4-kube-api-access-jnp9n" (OuterVolumeSpecName: "kube-api-access-jnp9n") pod "20a7f7d5-09a6-45e3-8af1-ac5be6efefe4" (UID: "20a7f7d5-09a6-45e3-8af1-ac5be6efefe4"). InnerVolumeSpecName "kube-api-access-jnp9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:32:20.818895 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.818873 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a7f7d5-09a6-45e3-8af1-ac5be6efefe4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "20a7f7d5-09a6-45e3-8af1-ac5be6efefe4" (UID: "20a7f7d5-09a6-45e3-8af1-ac5be6efefe4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:32:20.825215 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.825196 2581 scope.go:117] "RemoveContainer" containerID="1ded7ccb84c34c385b5f775b99ddaa89d34f9d1acb7dd08d59acdff1904233c3" Jul 06 21:32:20.825491 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:32:20.825471 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ded7ccb84c34c385b5f775b99ddaa89d34f9d1acb7dd08d59acdff1904233c3\": container with ID starting with 1ded7ccb84c34c385b5f775b99ddaa89d34f9d1acb7dd08d59acdff1904233c3 not found: ID does not exist" containerID="1ded7ccb84c34c385b5f775b99ddaa89d34f9d1acb7dd08d59acdff1904233c3" Jul 06 21:32:20.825538 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.825498 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ded7ccb84c34c385b5f775b99ddaa89d34f9d1acb7dd08d59acdff1904233c3"} err="failed to get container status \"1ded7ccb84c34c385b5f775b99ddaa89d34f9d1acb7dd08d59acdff1904233c3\": rpc error: code = NotFound desc = could not find container \"1ded7ccb84c34c385b5f775b99ddaa89d34f9d1acb7dd08d59acdff1904233c3\": container with ID starting with 1ded7ccb84c34c385b5f775b99ddaa89d34f9d1acb7dd08d59acdff1904233c3 not found: ID does not exist" Jul 06 21:32:20.825538 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.825514 2581 scope.go:117] "RemoveContainer" containerID="58162ea6ce4503034a1cc48097c9e28938bd055315622fe784b30debdbf2d2ef" Jul 06 21:32:20.825750 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:32:20.825733 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58162ea6ce4503034a1cc48097c9e28938bd055315622fe784b30debdbf2d2ef\": container with ID starting with 58162ea6ce4503034a1cc48097c9e28938bd055315622fe784b30debdbf2d2ef not found: ID does not exist" containerID="58162ea6ce4503034a1cc48097c9e28938bd055315622fe784b30debdbf2d2ef" Jul 06 21:32:20.825802 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.825756 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58162ea6ce4503034a1cc48097c9e28938bd055315622fe784b30debdbf2d2ef"} err="failed to get container status \"58162ea6ce4503034a1cc48097c9e28938bd055315622fe784b30debdbf2d2ef\": rpc error: code = NotFound desc = could not find container \"58162ea6ce4503034a1cc48097c9e28938bd055315622fe784b30debdbf2d2ef\": container with ID starting with 58162ea6ce4503034a1cc48097c9e28938bd055315622fe784b30debdbf2d2ef not found: ID does not exist" Jul 06 21:32:20.917997 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.917953 2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a771b291-e691-457b-8a8e-c63eff33a806-proxy-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:32:20.917997 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.917993 2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20a7f7d5-09a6-45e3-8af1-ac5be6efefe4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:32:20.917997 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.918004 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnp9n\" (UniqueName: \"kubernetes.io/projected/20a7f7d5-09a6-45e3-8af1-ac5be6efefe4-kube-api-access-jnp9n\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:32:20.918216 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.918012 2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20a7f7d5-09a6-45e3-8af1-ac5be6efefe4-proxy-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:32:20.918216 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.918021 2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a771b291-e691-457b-8a8e-c63eff33a806-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:32:20.918216 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:20.918029 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2bk8\" (UniqueName: \"kubernetes.io/projected/a771b291-e691-457b-8a8e-c63eff33a806-kube-api-access-r2bk8\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:32:21.118712 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:21.118684 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h"] Jul 06 21:32:21.125398 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:21.125373 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0aa94-predictor-b4bbfd678-9fc5h"] Jul 06 21:32:21.136853 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:21.136819 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm"] Jul 06 21:32:21.140542 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:21.140521 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0aa94-predictor-74dc6456dc-gnpkm"] Jul 06 21:32:21.786805 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:21.786712 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20a7f7d5-09a6-45e3-8af1-ac5be6efefe4" path="/var/lib/kubelet/pods/20a7f7d5-09a6-45e3-8af1-ac5be6efefe4/volumes" Jul 06 21:32:21.787226 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:21.787206 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a771b291-e691-457b-8a8e-c63eff33a806" path="/var/lib/kubelet/pods/a771b291-e691-457b-8a8e-c63eff33a806/volumes" Jul 06 21:32:23.787971 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:23.787941 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" Jul 06 21:32:23.788560 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:23.788533 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" podUID="fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Jul 06 21:32:24.790964 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:24.790936 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" Jul 06 21:32:24.791433 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:24.791405 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" podUID="412725d6-83f6-4641-9885-765577a11eaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Jul 06 21:32:33.788903 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:33.788866 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" podUID="fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Jul 06 21:32:34.791669 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:34.791626 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" podUID="412725d6-83f6-4641-9885-765577a11eaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Jul 06 21:32:43.789050 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:43.789014 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" podUID="fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Jul 06 21:32:44.791889 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:44.791853 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" podUID="412725d6-83f6-4641-9885-765577a11eaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Jul 06 21:32:53.789424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:53.789383 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" podUID="fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Jul 06 21:32:54.792208 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:32:54.792170 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" podUID="412725d6-83f6-4641-9885-765577a11eaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Jul 06 21:33:03.789935 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:03.789905 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" Jul 06 21:33:04.792245 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:04.792214 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" Jul 06 21:33:16.722543 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.722506 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl"] Jul 06 21:33:16.722971 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.722714 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" podUID="cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759" containerName="kserve-container" containerID="cri-o://9585a896e2b56dd50326f0230d89d2f62495aa20f2c9854e2a2d073bc389ea25" gracePeriod=30 Jul 06 21:33:16.722971 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.722767 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" podUID="cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759" containerName="kube-rbac-proxy" containerID="cri-o://608b2bbbd85ef911703e221615c42521476f724f9ebb7192396d09b6a654155b" gracePeriod=30 Jul 06 21:33:16.802176 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.802139 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6"] Jul 06 21:33:16.802532 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.802520 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a7f7d5-09a6-45e3-8af1-ac5be6efefe4" containerName="kube-rbac-proxy" Jul 06 21:33:16.802587 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.802534 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a7f7d5-09a6-45e3-8af1-ac5be6efefe4" containerName="kube-rbac-proxy" Jul 06 21:33:16.802587 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.802548 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a771b291-e691-457b-8a8e-c63eff33a806" containerName="kube-rbac-proxy" Jul 06 21:33:16.802587 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.802554 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a771b291-e691-457b-8a8e-c63eff33a806" containerName="kube-rbac-proxy" Jul 06 21:33:16.802587 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.802561 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a7f7d5-09a6-45e3-8af1-ac5be6efefe4" containerName="kserve-container" Jul 06 21:33:16.802587 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.802567 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a7f7d5-09a6-45e3-8af1-ac5be6efefe4" containerName="kserve-container" Jul 06 21:33:16.802587 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.802579 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a771b291-e691-457b-8a8e-c63eff33a806" containerName="kserve-container" Jul 06 21:33:16.802587 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.802585 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a771b291-e691-457b-8a8e-c63eff33a806" containerName="kserve-container" Jul 06 21:33:16.802813 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.802639 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a771b291-e691-457b-8a8e-c63eff33a806" containerName="kube-rbac-proxy" Jul 06 21:33:16.802813 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.802648 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a771b291-e691-457b-8a8e-c63eff33a806" containerName="kserve-container" Jul 06 21:33:16.802813 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.802656 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a7f7d5-09a6-45e3-8af1-ac5be6efefe4" containerName="kserve-container" Jul 06 21:33:16.802813 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.802661 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a7f7d5-09a6-45e3-8af1-ac5be6efefe4" containerName="kube-rbac-proxy" Jul 06 21:33:16.807089 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.807069 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" Jul 06 21:33:16.809952 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.809926 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-599ee-kube-rbac-proxy-sar-config\"" Jul 06 21:33:16.810459 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.810437 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-599ee-predictor-serving-cert\"" Jul 06 21:33:16.815924 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.815834 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6"] Jul 06 21:33:16.843128 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.843080 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh"] Jul 06 21:33:16.843467 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.843404 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" podUID="0b6edf03-b1e4-4689-84bc-4d973357b579" containerName="kserve-container" containerID="cri-o://7aff816040954ef637c49f45dd1236d32baa0e845c2ec1d7c24c9d4599f9e09a" gracePeriod=30 Jul 06 21:33:16.843467 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.843432 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" podUID="0b6edf03-b1e4-4689-84bc-4d973357b579" containerName="kube-rbac-proxy" containerID="cri-o://51b6d9545436d67a6a9335193c6ec16379a3bc835daaa62a18f8f7d01e8c3d8e" gracePeriod=30 Jul 06 21:33:16.901178 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.901128 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db17289d-5948-4e9e-a0c1-481806859036-proxy-tls\") pod \"success-200-isvc-599ee-predictor-58f56bd85d-96fm6\" (UID: \"db17289d-5948-4e9e-a0c1-481806859036\") " pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" Jul 06 21:33:16.901178 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.901184 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db17289d-5948-4e9e-a0c1-481806859036-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-599ee-predictor-58f56bd85d-96fm6\" (UID: \"db17289d-5948-4e9e-a0c1-481806859036\") " pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" Jul 06 21:33:16.901438 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.901366 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gx4w\" (UniqueName: \"kubernetes.io/projected/db17289d-5948-4e9e-a0c1-481806859036-kube-api-access-5gx4w\") pod \"success-200-isvc-599ee-predictor-58f56bd85d-96fm6\" (UID: \"db17289d-5948-4e9e-a0c1-481806859036\") " pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" Jul 06 21:33:16.906571 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.906540 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst"] Jul 06 21:33:16.910140 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.910118 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" Jul 06 21:33:16.913053 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.913027 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-599ee-predictor-serving-cert\"" Jul 06 21:33:16.913161 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.913132 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-599ee-kube-rbac-proxy-sar-config\"" Jul 06 21:33:16.920970 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:16.920947 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst"] Jul 06 21:33:17.002471 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.002365 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db17289d-5948-4e9e-a0c1-481806859036-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-599ee-predictor-58f56bd85d-96fm6\" (UID: \"db17289d-5948-4e9e-a0c1-481806859036\") " pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" Jul 06 21:33:17.002647 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.002477 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4186ac0-8140-41eb-893d-b5417ec52a8f-proxy-tls\") pod \"error-404-isvc-599ee-predictor-6fdd64d5bc-nplst\" (UID: \"a4186ac0-8140-41eb-893d-b5417ec52a8f\") " pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" Jul 06 21:33:17.002647 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.002568 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4186ac0-8140-41eb-893d-b5417ec52a8f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-599ee-predictor-6fdd64d5bc-nplst\" (UID: \"a4186ac0-8140-41eb-893d-b5417ec52a8f\") " pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" Jul 06 21:33:17.002813 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.002787 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gx4w\" (UniqueName: \"kubernetes.io/projected/db17289d-5948-4e9e-a0c1-481806859036-kube-api-access-5gx4w\") pod \"success-200-isvc-599ee-predictor-58f56bd85d-96fm6\" (UID: \"db17289d-5948-4e9e-a0c1-481806859036\") " pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" Jul 06 21:33:17.002881 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.002829 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84sfd\" (UniqueName: \"kubernetes.io/projected/a4186ac0-8140-41eb-893d-b5417ec52a8f-kube-api-access-84sfd\") pod \"error-404-isvc-599ee-predictor-6fdd64d5bc-nplst\" (UID: \"a4186ac0-8140-41eb-893d-b5417ec52a8f\") " pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" Jul 06 21:33:17.002881 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.002872 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db17289d-5948-4e9e-a0c1-481806859036-proxy-tls\") pod \"success-200-isvc-599ee-predictor-58f56bd85d-96fm6\" (UID: \"db17289d-5948-4e9e-a0c1-481806859036\") " pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" Jul 06 21:33:17.002980 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:33:17.002965 2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-599ee-predictor-serving-cert: secret "success-200-isvc-599ee-predictor-serving-cert" not found Jul 06 21:33:17.003028 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:33:17.003012 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db17289d-5948-4e9e-a0c1-481806859036-proxy-tls podName:db17289d-5948-4e9e-a0c1-481806859036 nodeName:}" failed. No retries permitted until 2026-07-06 21:33:17.502999152 +0000 UTC m=+1222.265912134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/db17289d-5948-4e9e-a0c1-481806859036-proxy-tls") pod "success-200-isvc-599ee-predictor-58f56bd85d-96fm6" (UID: "db17289d-5948-4e9e-a0c1-481806859036") : secret "success-200-isvc-599ee-predictor-serving-cert" not found Jul 06 21:33:17.003172 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.003151 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db17289d-5948-4e9e-a0c1-481806859036-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-599ee-predictor-58f56bd85d-96fm6\" (UID: \"db17289d-5948-4e9e-a0c1-481806859036\") " pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" Jul 06 21:33:17.019310 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.019283 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gx4w\" (UniqueName: \"kubernetes.io/projected/db17289d-5948-4e9e-a0c1-481806859036-kube-api-access-5gx4w\") pod \"success-200-isvc-599ee-predictor-58f56bd85d-96fm6\" (UID: \"db17289d-5948-4e9e-a0c1-481806859036\") " pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" Jul 06 21:33:17.062564 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.062531 2581 generic.go:356] "Generic (PLEG): container finished" podID="cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759" containerID="608b2bbbd85ef911703e221615c42521476f724f9ebb7192396d09b6a654155b" exitCode=2 Jul 06 21:33:17.062739 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.062611 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" event={"ID":"cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759","Type":"ContainerDied","Data":"608b2bbbd85ef911703e221615c42521476f724f9ebb7192396d09b6a654155b"} Jul 06 21:33:17.064306 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.064284 2581 generic.go:356] "Generic (PLEG): container finished" podID="0b6edf03-b1e4-4689-84bc-4d973357b579" containerID="51b6d9545436d67a6a9335193c6ec16379a3bc835daaa62a18f8f7d01e8c3d8e" exitCode=2 Jul 06 21:33:17.064466 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.064360 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" event={"ID":"0b6edf03-b1e4-4689-84bc-4d973357b579","Type":"ContainerDied","Data":"51b6d9545436d67a6a9335193c6ec16379a3bc835daaa62a18f8f7d01e8c3d8e"} Jul 06 21:33:17.103529 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.103500 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4186ac0-8140-41eb-893d-b5417ec52a8f-proxy-tls\") pod \"error-404-isvc-599ee-predictor-6fdd64d5bc-nplst\" (UID: \"a4186ac0-8140-41eb-893d-b5417ec52a8f\") " pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" Jul 06 21:33:17.103674 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.103559 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4186ac0-8140-41eb-893d-b5417ec52a8f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-599ee-predictor-6fdd64d5bc-nplst\" (UID: \"a4186ac0-8140-41eb-893d-b5417ec52a8f\") " pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" Jul 06 21:33:17.103674 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.103597 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84sfd\" (UniqueName: \"kubernetes.io/projected/a4186ac0-8140-41eb-893d-b5417ec52a8f-kube-api-access-84sfd\") pod \"error-404-isvc-599ee-predictor-6fdd64d5bc-nplst\" (UID: \"a4186ac0-8140-41eb-893d-b5417ec52a8f\") " pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" Jul 06 21:33:17.104242 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.104222 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4186ac0-8140-41eb-893d-b5417ec52a8f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-599ee-predictor-6fdd64d5bc-nplst\" (UID: \"a4186ac0-8140-41eb-893d-b5417ec52a8f\") " pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" Jul 06 21:33:17.105972 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.105951 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4186ac0-8140-41eb-893d-b5417ec52a8f-proxy-tls\") pod \"error-404-isvc-599ee-predictor-6fdd64d5bc-nplst\" (UID: \"a4186ac0-8140-41eb-893d-b5417ec52a8f\") " pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" Jul 06 21:33:17.112906 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.112880 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84sfd\" (UniqueName: \"kubernetes.io/projected/a4186ac0-8140-41eb-893d-b5417ec52a8f-kube-api-access-84sfd\") pod \"error-404-isvc-599ee-predictor-6fdd64d5bc-nplst\" (UID: \"a4186ac0-8140-41eb-893d-b5417ec52a8f\") " pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" Jul 06 21:33:17.220485 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.220446 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" Jul 06 21:33:17.359415 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.359383 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst"] Jul 06 21:33:17.362860 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:33:17.362835 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4186ac0_8140_41eb_893d_b5417ec52a8f.slice/crio-99d133df8e278bc9e7f400e142fc14305cd9d908ecec487bc12600033991cc63 WatchSource:0}: Error finding container 99d133df8e278bc9e7f400e142fc14305cd9d908ecec487bc12600033991cc63: Status 404 returned error can't find the container with id 99d133df8e278bc9e7f400e142fc14305cd9d908ecec487bc12600033991cc63 Jul 06 21:33:17.507303 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.507271 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db17289d-5948-4e9e-a0c1-481806859036-proxy-tls\") pod \"success-200-isvc-599ee-predictor-58f56bd85d-96fm6\" (UID: \"db17289d-5948-4e9e-a0c1-481806859036\") " pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" Jul 06 21:33:17.509549 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.509530 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db17289d-5948-4e9e-a0c1-481806859036-proxy-tls\") pod \"success-200-isvc-599ee-predictor-58f56bd85d-96fm6\" (UID: \"db17289d-5948-4e9e-a0c1-481806859036\") " pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" Jul 06 21:33:17.720743 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.720691 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" Jul 06 21:33:17.855392 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:17.855311 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6"] Jul 06 21:33:17.857617 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:33:17.857585 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb17289d_5948_4e9e_a0c1_481806859036.slice/crio-d4d60098fa62e9a35d9753c61b89948c856d5a601a8c5930b4f91b9bcd6eda55 WatchSource:0}: Error finding container d4d60098fa62e9a35d9753c61b89948c856d5a601a8c5930b4f91b9bcd6eda55: Status 404 returned error can't find the container with id d4d60098fa62e9a35d9753c61b89948c856d5a601a8c5930b4f91b9bcd6eda55 Jul 06 21:33:18.071006 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:18.070908 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" event={"ID":"a4186ac0-8140-41eb-893d-b5417ec52a8f","Type":"ContainerStarted","Data":"b42620d28f89b5820ac56099bfa14a279dd9d893d9e32a346dcf3eaaf9634a70"} Jul 06 21:33:18.071006 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:18.070981 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" event={"ID":"a4186ac0-8140-41eb-893d-b5417ec52a8f","Type":"ContainerStarted","Data":"9d5bc6c4f56f280c56d5a1b09d267259f6e852596074dac3230dee584b735212"} Jul 06 21:33:18.071234 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:18.071014 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" event={"ID":"a4186ac0-8140-41eb-893d-b5417ec52a8f","Type":"ContainerStarted","Data":"99d133df8e278bc9e7f400e142fc14305cd9d908ecec487bc12600033991cc63"} Jul 06 21:33:18.071234 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:18.071046 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" Jul 06 21:33:18.071234 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:18.071075 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" Jul 06 21:33:18.072832 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:18.072807 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" event={"ID":"db17289d-5948-4e9e-a0c1-481806859036","Type":"ContainerStarted","Data":"b4311b586c617e397650e0e76c45ddbfa87de1114396b20d6a1505d49379aa86"} Jul 06 21:33:18.072929 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:18.072853 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" event={"ID":"db17289d-5948-4e9e-a0c1-481806859036","Type":"ContainerStarted","Data":"047453869c6ef0f79b43c22347d236dc166ce8dd9f81498e8baa6b85a85b168b"} Jul 06 21:33:18.072929 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:18.072866 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" podUID="a4186ac0-8140-41eb-893d-b5417ec52a8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Jul 06 21:33:18.072929 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:18.072885 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" Jul 06 21:33:18.072929 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:18.072905 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" event={"ID":"db17289d-5948-4e9e-a0c1-481806859036","Type":"ContainerStarted","Data":"d4d60098fa62e9a35d9753c61b89948c856d5a601a8c5930b4f91b9bcd6eda55"} Jul 06 21:33:18.092477 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:18.092434 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" podStartSLOduration=2.092419637 podStartE2EDuration="2.092419637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:33:18.091991628 +0000 UTC m=+1222.854904633" watchObservedRunningTime="2026-07-06 21:33:18.092419637 +0000 UTC m=+1222.855332640" Jul 06 21:33:18.111992 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:18.111938 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" podStartSLOduration=2.111923674 podStartE2EDuration="2.111923674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:33:18.110265392 +0000 UTC m=+1222.873178409" watchObservedRunningTime="2026-07-06 21:33:18.111923674 +0000 UTC m=+1222.874836677" Jul 06 21:33:19.078256 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:19.078216 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" podUID="a4186ac0-8140-41eb-893d-b5417ec52a8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Jul 06 21:33:19.078748 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:19.078377 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" Jul 06 21:33:19.079572 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:19.079550 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" podUID="db17289d-5948-4e9e-a0c1-481806859036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Jul 06 21:33:19.708548 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:19.708507 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" podUID="cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused" Jul 06 21:33:19.713946 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:19.713918 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" podUID="cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Jul 06 21:33:20.085082 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.085050 2581 generic.go:356] "Generic (PLEG): container finished" podID="cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759" containerID="9585a896e2b56dd50326f0230d89d2f62495aa20f2c9854e2a2d073bc389ea25" exitCode=0 Jul 06 21:33:20.085559 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.085129 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" event={"ID":"cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759","Type":"ContainerDied","Data":"9585a896e2b56dd50326f0230d89d2f62495aa20f2c9854e2a2d073bc389ea25"} Jul 06 21:33:20.085767 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.085740 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" podUID="db17289d-5948-4e9e-a0c1-481806859036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Jul 06 21:33:20.173172 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.173143 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" Jul 06 21:33:20.292901 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.292877 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" Jul 06 21:33:20.333165 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.333137 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759-proxy-tls\") pod \"cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759\" (UID: \"cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759\") " Jul 06 21:33:20.333365 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.333204 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrdd7\" (UniqueName: \"kubernetes.io/projected/cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759-kube-api-access-lrdd7\") pod \"cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759\" (UID: \"cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759\") " Jul 06 21:33:20.333365 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.333277 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759-kube-rbac-proxy-sar-config\") pod \"cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759\" (UID: \"cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759\") " Jul 06 21:33:20.333633 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.333606 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759" (UID: "cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:33:20.335356 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.335314 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759" (UID: "cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:33:20.335416 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.335392 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759-kube-api-access-lrdd7" (OuterVolumeSpecName: "kube-api-access-lrdd7") pod "cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759" (UID: "cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759"). InnerVolumeSpecName "kube-api-access-lrdd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:33:20.434221 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.434183 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ltkr\" (UniqueName: \"kubernetes.io/projected/0b6edf03-b1e4-4689-84bc-4d973357b579-kube-api-access-4ltkr\") pod \"0b6edf03-b1e4-4689-84bc-4d973357b579\" (UID: \"0b6edf03-b1e4-4689-84bc-4d973357b579\") " Jul 06 21:33:20.434434 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.434242 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b6edf03-b1e4-4689-84bc-4d973357b579-proxy-tls\") pod \"0b6edf03-b1e4-4689-84bc-4d973357b579\" (UID: \"0b6edf03-b1e4-4689-84bc-4d973357b579\") " Jul 06 21:33:20.434434 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.434302 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b6edf03-b1e4-4689-84bc-4d973357b579-kube-rbac-proxy-sar-config\") pod \"0b6edf03-b1e4-4689-84bc-4d973357b579\" (UID: \"0b6edf03-b1e4-4689-84bc-4d973357b579\") " Jul 06 21:33:20.434635 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.434612 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrdd7\" (UniqueName: \"kubernetes.io/projected/cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759-kube-api-access-lrdd7\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:33:20.434682 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.434642 2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:33:20.434682 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.434657 2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759-proxy-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:33:20.434744 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.434674 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6edf03-b1e4-4689-84bc-4d973357b579-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0b6edf03-b1e4-4689-84bc-4d973357b579" (UID: "0b6edf03-b1e4-4689-84bc-4d973357b579"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:33:20.436436 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.436412 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6edf03-b1e4-4689-84bc-4d973357b579-kube-api-access-4ltkr" (OuterVolumeSpecName: "kube-api-access-4ltkr") pod "0b6edf03-b1e4-4689-84bc-4d973357b579" (UID: "0b6edf03-b1e4-4689-84bc-4d973357b579"). InnerVolumeSpecName "kube-api-access-4ltkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:33:20.436503 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.436443 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b6edf03-b1e4-4689-84bc-4d973357b579-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0b6edf03-b1e4-4689-84bc-4d973357b579" (UID: "0b6edf03-b1e4-4689-84bc-4d973357b579"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:33:20.536138 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.536044 2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b6edf03-b1e4-4689-84bc-4d973357b579-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:33:20.536138 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.536083 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ltkr\" (UniqueName: \"kubernetes.io/projected/0b6edf03-b1e4-4689-84bc-4d973357b579-kube-api-access-4ltkr\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:33:20.536138 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:20.536093 2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b6edf03-b1e4-4689-84bc-4d973357b579-proxy-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:33:21.091646 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:21.091608 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" event={"ID":"cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759","Type":"ContainerDied","Data":"0db5581e30488b98cf0056c1aa88cfce04a8f1f286a589a683f1dec7c0602855"} Jul 06 21:33:21.092063 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:21.091677 2581 scope.go:117] "RemoveContainer" containerID="608b2bbbd85ef911703e221615c42521476f724f9ebb7192396d09b6a654155b" Jul 06 21:33:21.092063 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:21.091622 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl" Jul 06 21:33:21.093299 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:21.093253 2581 generic.go:356] "Generic (PLEG): container finished" podID="0b6edf03-b1e4-4689-84bc-4d973357b579" containerID="7aff816040954ef637c49f45dd1236d32baa0e845c2ec1d7c24c9d4599f9e09a" exitCode=0 Jul 06 21:33:21.093426 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:21.093298 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" event={"ID":"0b6edf03-b1e4-4689-84bc-4d973357b579","Type":"ContainerDied","Data":"7aff816040954ef637c49f45dd1236d32baa0e845c2ec1d7c24c9d4599f9e09a"} Jul 06 21:33:21.093426 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:21.093330 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" Jul 06 21:33:21.093426 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:21.093362 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh" event={"ID":"0b6edf03-b1e4-4689-84bc-4d973357b579","Type":"ContainerDied","Data":"15b61d63538ceebd549758c8634e907677b4773da6705c3014cb2027ace0b94a"} Jul 06 21:33:21.099865 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:21.099846 2581 scope.go:117] "RemoveContainer" containerID="9585a896e2b56dd50326f0230d89d2f62495aa20f2c9854e2a2d073bc389ea25" Jul 06 21:33:21.108329 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:21.108312 2581 scope.go:117] "RemoveContainer" containerID="51b6d9545436d67a6a9335193c6ec16379a3bc835daaa62a18f8f7d01e8c3d8e" Jul 06 21:33:21.115629 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:21.115609 2581 scope.go:117] "RemoveContainer" containerID="7aff816040954ef637c49f45dd1236d32baa0e845c2ec1d7c24c9d4599f9e09a" Jul 06 21:33:21.118710 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:21.118684 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl"] Jul 06 21:33:21.122914 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:21.122890 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-22b2a-predictor-7b88574587-dq2jl"] Jul 06 21:33:21.124086 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:21.124068 2581 scope.go:117] "RemoveContainer" containerID="51b6d9545436d67a6a9335193c6ec16379a3bc835daaa62a18f8f7d01e8c3d8e" Jul 06 21:33:21.124399 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:33:21.124382 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51b6d9545436d67a6a9335193c6ec16379a3bc835daaa62a18f8f7d01e8c3d8e\": container with ID starting with 51b6d9545436d67a6a9335193c6ec16379a3bc835daaa62a18f8f7d01e8c3d8e not found: ID does not exist" containerID="51b6d9545436d67a6a9335193c6ec16379a3bc835daaa62a18f8f7d01e8c3d8e" Jul 06 21:33:21.124452 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:21.124411 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51b6d9545436d67a6a9335193c6ec16379a3bc835daaa62a18f8f7d01e8c3d8e"} err="failed to get container status \"51b6d9545436d67a6a9335193c6ec16379a3bc835daaa62a18f8f7d01e8c3d8e\": rpc error: code = NotFound desc = could not find container \"51b6d9545436d67a6a9335193c6ec16379a3bc835daaa62a18f8f7d01e8c3d8e\": container with ID starting with 51b6d9545436d67a6a9335193c6ec16379a3bc835daaa62a18f8f7d01e8c3d8e not found: ID does not exist" Jul 06 21:33:21.124452 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:21.124431 2581 scope.go:117] "RemoveContainer" containerID="7aff816040954ef637c49f45dd1236d32baa0e845c2ec1d7c24c9d4599f9e09a" Jul 06 21:33:21.124701 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:33:21.124683 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aff816040954ef637c49f45dd1236d32baa0e845c2ec1d7c24c9d4599f9e09a\": container with ID starting with 7aff816040954ef637c49f45dd1236d32baa0e845c2ec1d7c24c9d4599f9e09a not found: ID does not exist" containerID="7aff816040954ef637c49f45dd1236d32baa0e845c2ec1d7c24c9d4599f9e09a" Jul 06 21:33:21.124765 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:21.124707 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aff816040954ef637c49f45dd1236d32baa0e845c2ec1d7c24c9d4599f9e09a"} err="failed to get container status \"7aff816040954ef637c49f45dd1236d32baa0e845c2ec1d7c24c9d4599f9e09a\": rpc error: code = NotFound desc = could not find container \"7aff816040954ef637c49f45dd1236d32baa0e845c2ec1d7c24c9d4599f9e09a\": container with ID starting with 7aff816040954ef637c49f45dd1236d32baa0e845c2ec1d7c24c9d4599f9e09a not found: ID does not exist" Jul 06 21:33:21.135466 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:21.135440 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh"] Jul 06 21:33:21.141563 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:21.141540 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-22b2a-predictor-66499f9fc4-mklrh"] Jul 06 21:33:21.786716 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:21.786679 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b6edf03-b1e4-4689-84bc-4d973357b579" path="/var/lib/kubelet/pods/0b6edf03-b1e4-4689-84bc-4d973357b579/volumes" Jul 06 21:33:21.787105 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:21.787092 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759" path="/var/lib/kubelet/pods/cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759/volumes" Jul 06 21:33:24.083105 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:24.083073 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" Jul 06 21:33:24.083542 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:24.083519 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" podUID="a4186ac0-8140-41eb-893d-b5417ec52a8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Jul 06 21:33:25.090531 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:25.090501 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" Jul 06 21:33:25.090904 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:25.090880 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" podUID="db17289d-5948-4e9e-a0c1-481806859036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Jul 06 21:33:27.103081 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.103049 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch"] Jul 06 21:33:27.103742 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.103705 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" podUID="fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc" containerName="kserve-container" containerID="cri-o://32e3ff29fa06739d765f1dff8f49472a5479a56fd80b69312b24f085a718dc83" gracePeriod=30 Jul 06 21:33:27.104698 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.104646 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" podUID="fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc" containerName="kube-rbac-proxy" containerID="cri-o://0d805541d7de269b7a61cba6036f70f23eaa551a4e9c98e8a2b9510164e619b8" gracePeriod=30 Jul 06 21:33:27.114814 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.114172 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp"] Jul 06 21:33:27.114814 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.114754 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759" containerName="kube-rbac-proxy" Jul 06 21:33:27.114814 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.114771 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759" containerName="kube-rbac-proxy" Jul 06 21:33:27.114814 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.114799 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759" containerName="kserve-container" Jul 06 21:33:27.114814 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.114808 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759" containerName="kserve-container" Jul 06 21:33:27.114814 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.114816 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b6edf03-b1e4-4689-84bc-4d973357b579" containerName="kserve-container" Jul 06 21:33:27.115125 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.114825 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b6edf03-b1e4-4689-84bc-4d973357b579" containerName="kserve-container" Jul 06 21:33:27.115125 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.114851 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b6edf03-b1e4-4689-84bc-4d973357b579" containerName="kube-rbac-proxy" Jul 06 21:33:27.115125 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.114859 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b6edf03-b1e4-4689-84bc-4d973357b579" containerName="kube-rbac-proxy" Jul 06 21:33:27.115125 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.114941 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759" containerName="kube-rbac-proxy" Jul 06 21:33:27.115125 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.114953 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b6edf03-b1e4-4689-84bc-4d973357b579" containerName="kserve-container" Jul 06 21:33:27.115125 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.114967 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbbf9c58-3bf6-4d8f-9367-5cb1b4ba7759" containerName="kserve-container" Jul 06 21:33:27.115125 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.114979 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b6edf03-b1e4-4689-84bc-4d973357b579" containerName="kube-rbac-proxy" Jul 06 21:33:27.121961 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.121844 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" Jul 06 21:33:27.124276 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.124258 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9ded0-predictor-serving-cert\"" Jul 06 21:33:27.124632 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.124615 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9ded0-kube-rbac-proxy-sar-config\"" Jul 06 21:33:27.130443 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.130416 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp"] Jul 06 21:33:27.222170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.222138 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx"] Jul 06 21:33:27.225788 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.225766 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" Jul 06 21:33:27.228232 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.228207 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9ded0-predictor-serving-cert\"" Jul 06 21:33:27.228510 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.228488 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9ded0-kube-rbac-proxy-sar-config\"" Jul 06 21:33:27.236887 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.236863 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx"] Jul 06 21:33:27.276373 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.276328 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg"] Jul 06 21:33:27.276618 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.276590 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" podUID="412725d6-83f6-4641-9885-765577a11eaf" containerName="kserve-container" containerID="cri-o://7795dc6430bc773a4d8bb995d5cd73ccb0c685cbc50b2e3b8ddd77385df36bab" gracePeriod=30 Jul 06 21:33:27.276702 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.276604 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" podUID="412725d6-83f6-4641-9885-765577a11eaf" containerName="kube-rbac-proxy" containerID="cri-o://32b87d25f8feb9e34247c10101714a484ae6fcbb2abf8ffed20a4f5b1f7c0804" gracePeriod=30 Jul 06 21:33:27.297803 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.297774 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9dc8783d-9353-4694-982e-febf4d48ee85-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp\" (UID: \"9dc8783d-9353-4694-982e-febf4d48ee85\") " pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" Jul 06 21:33:27.297898 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.297818 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9dc8783d-9353-4694-982e-febf4d48ee85-proxy-tls\") pod \"success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp\" (UID: \"9dc8783d-9353-4694-982e-febf4d48ee85\") " pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" Jul 06 21:33:27.297898 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.297882 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8djk\" (UniqueName: \"kubernetes.io/projected/9dc8783d-9353-4694-982e-febf4d48ee85-kube-api-access-q8djk\") pod \"success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp\" (UID: \"9dc8783d-9353-4694-982e-febf4d48ee85\") " pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" Jul 06 21:33:27.398670 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.398639 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/919fd4d5-49d6-4638-b938-4285e9cce70d-proxy-tls\") pod \"error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx\" (UID: \"919fd4d5-49d6-4638-b938-4285e9cce70d\") " pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" Jul 06 21:33:27.398812 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.398683 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/919fd4d5-49d6-4638-b938-4285e9cce70d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx\" (UID: \"919fd4d5-49d6-4638-b938-4285e9cce70d\") " pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" Jul 06 21:33:27.398812 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.398752 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9dc8783d-9353-4694-982e-febf4d48ee85-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp\" (UID: \"9dc8783d-9353-4694-982e-febf4d48ee85\") " pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" Jul 06 21:33:27.398899 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.398816 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9dc8783d-9353-4694-982e-febf4d48ee85-proxy-tls\") pod \"success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp\" (UID: \"9dc8783d-9353-4694-982e-febf4d48ee85\") " pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" Jul 06 21:33:27.398899 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.398846 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxkzb\" (UniqueName: \"kubernetes.io/projected/919fd4d5-49d6-4638-b938-4285e9cce70d-kube-api-access-sxkzb\") pod \"error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx\" (UID: \"919fd4d5-49d6-4638-b938-4285e9cce70d\") " pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" Jul 06 21:33:27.398899 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.398892 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8djk\" (UniqueName: \"kubernetes.io/projected/9dc8783d-9353-4694-982e-febf4d48ee85-kube-api-access-q8djk\") pod \"success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp\" (UID: \"9dc8783d-9353-4694-982e-febf4d48ee85\") " pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" Jul 06 21:33:27.399399 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.399377 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9dc8783d-9353-4694-982e-febf4d48ee85-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp\" (UID: \"9dc8783d-9353-4694-982e-febf4d48ee85\") " pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" Jul 06 21:33:27.401178 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.401156 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9dc8783d-9353-4694-982e-febf4d48ee85-proxy-tls\") pod \"success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp\" (UID: \"9dc8783d-9353-4694-982e-febf4d48ee85\") " pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" Jul 06 21:33:27.407752 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.407723 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8djk\" (UniqueName: \"kubernetes.io/projected/9dc8783d-9353-4694-982e-febf4d48ee85-kube-api-access-q8djk\") pod \"success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp\" (UID: \"9dc8783d-9353-4694-982e-febf4d48ee85\") " pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" Jul 06 21:33:27.433789 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.433754 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" Jul 06 21:33:27.501941 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.501898 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/919fd4d5-49d6-4638-b938-4285e9cce70d-proxy-tls\") pod \"error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx\" (UID: \"919fd4d5-49d6-4638-b938-4285e9cce70d\") " pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" Jul 06 21:33:27.502109 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.501963 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/919fd4d5-49d6-4638-b938-4285e9cce70d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx\" (UID: \"919fd4d5-49d6-4638-b938-4285e9cce70d\") " pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" Jul 06 21:33:27.502109 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.502027 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxkzb\" (UniqueName: \"kubernetes.io/projected/919fd4d5-49d6-4638-b938-4285e9cce70d-kube-api-access-sxkzb\") pod \"error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx\" (UID: \"919fd4d5-49d6-4638-b938-4285e9cce70d\") " pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" Jul 06 21:33:27.502582 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:33:27.502553 2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-serving-cert: secret "error-404-isvc-9ded0-predictor-serving-cert" not found Jul 06 21:33:27.502694 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:33:27.502645 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/919fd4d5-49d6-4638-b938-4285e9cce70d-proxy-tls podName:919fd4d5-49d6-4638-b938-4285e9cce70d nodeName:}" failed. No retries permitted until 2026-07-06 21:33:28.002622963 +0000 UTC m=+1232.765535962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/919fd4d5-49d6-4638-b938-4285e9cce70d-proxy-tls") pod "error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" (UID: "919fd4d5-49d6-4638-b938-4285e9cce70d") : secret "error-404-isvc-9ded0-predictor-serving-cert" not found Jul 06 21:33:27.503615 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.503585 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/919fd4d5-49d6-4638-b938-4285e9cce70d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx\" (UID: \"919fd4d5-49d6-4638-b938-4285e9cce70d\") " pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" Jul 06 21:33:27.518514 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.518359 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxkzb\" (UniqueName: \"kubernetes.io/projected/919fd4d5-49d6-4638-b938-4285e9cce70d-kube-api-access-sxkzb\") pod \"error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx\" (UID: \"919fd4d5-49d6-4638-b938-4285e9cce70d\") " pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" Jul 06 21:33:27.607939 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:27.607887 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp"] Jul 06 21:33:28.006846 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:28.006750 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/919fd4d5-49d6-4638-b938-4285e9cce70d-proxy-tls\") pod \"error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx\" (UID: \"919fd4d5-49d6-4638-b938-4285e9cce70d\") " pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" Jul 06 21:33:28.009152 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:28.009128 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/919fd4d5-49d6-4638-b938-4285e9cce70d-proxy-tls\") pod \"error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx\" (UID: \"919fd4d5-49d6-4638-b938-4285e9cce70d\") " pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" Jul 06 21:33:28.130330 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:28.130298 2581 generic.go:356] "Generic (PLEG): container finished" podID="412725d6-83f6-4641-9885-765577a11eaf" containerID="32b87d25f8feb9e34247c10101714a484ae6fcbb2abf8ffed20a4f5b1f7c0804" exitCode=2 Jul 06 21:33:28.130803 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:28.130369 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" event={"ID":"412725d6-83f6-4641-9885-765577a11eaf","Type":"ContainerDied","Data":"32b87d25f8feb9e34247c10101714a484ae6fcbb2abf8ffed20a4f5b1f7c0804"} Jul 06 21:33:28.132057 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:28.132036 2581 generic.go:356] "Generic (PLEG): container finished" podID="fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc" containerID="0d805541d7de269b7a61cba6036f70f23eaa551a4e9c98e8a2b9510164e619b8" exitCode=2 Jul 06 21:33:28.132162 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:28.132110 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" event={"ID":"fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc","Type":"ContainerDied","Data":"0d805541d7de269b7a61cba6036f70f23eaa551a4e9c98e8a2b9510164e619b8"} Jul 06 21:33:28.133605 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:28.133582 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" event={"ID":"9dc8783d-9353-4694-982e-febf4d48ee85","Type":"ContainerStarted","Data":"9720f2a5dd1ab6342980ae39a333ed875f3c2032199c73d2d06a8b7edd4fc0e3"} Jul 06 21:33:28.133679 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:28.133634 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" event={"ID":"9dc8783d-9353-4694-982e-febf4d48ee85","Type":"ContainerStarted","Data":"dc5d88742458cb07440b471f2241d7619fac6180864125e9938dc6da5b805497"} Jul 06 21:33:28.133679 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:28.133661 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" Jul 06 21:33:28.133746 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:28.133682 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" event={"ID":"9dc8783d-9353-4694-982e-febf4d48ee85","Type":"ContainerStarted","Data":"c7bcde0cbcd946532b338f186cfcb3259f2152bced2f96f27be69d39bd812acf"} Jul 06 21:33:28.133746 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:28.133710 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" Jul 06 21:33:28.134954 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:28.134931 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" podUID="9dc8783d-9353-4694-982e-febf4d48ee85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Jul 06 21:33:28.137073 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:28.137058 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" Jul 06 21:33:28.160538 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:28.160492 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" podStartSLOduration=1.160475125 podStartE2EDuration="1.160475125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:33:28.158416816 +0000 UTC m=+1232.921329821" watchObservedRunningTime="2026-07-06 21:33:28.160475125 +0000 UTC m=+1232.923388129" Jul 06 21:33:28.275978 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:28.275898 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx"] Jul 06 21:33:28.278812 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:33:28.278784 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod919fd4d5_49d6_4638_b938_4285e9cce70d.slice/crio-e6f3440699f8dda1101bd97c0c415f035ea39c13a0c1abcf93bab7b2a60b587c WatchSource:0}: Error finding container e6f3440699f8dda1101bd97c0c415f035ea39c13a0c1abcf93bab7b2a60b587c: Status 404 returned error can't find the container with id e6f3440699f8dda1101bd97c0c415f035ea39c13a0c1abcf93bab7b2a60b587c Jul 06 21:33:28.781799 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:28.781752 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" podUID="fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused" Jul 06 21:33:29.140169 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:29.140134 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" event={"ID":"919fd4d5-49d6-4638-b938-4285e9cce70d","Type":"ContainerStarted","Data":"a3038ba06b576c81bcb8ec3d07c41490561cf0ae2bd12adce659f1481e04fd4e"} Jul 06 21:33:29.140628 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:29.140194 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" event={"ID":"919fd4d5-49d6-4638-b938-4285e9cce70d","Type":"ContainerStarted","Data":"c90c86031c7aefac581108b58bdffd1900ff461e47acee7d74b4fdbfa6e3b696"} Jul 06 21:33:29.140628 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:29.140445 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" Jul 06 21:33:29.140628 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:29.140504 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" Jul 06 21:33:29.140628 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:29.140445 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" podUID="9dc8783d-9353-4694-982e-febf4d48ee85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Jul 06 21:33:29.140628 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:29.140536 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" event={"ID":"919fd4d5-49d6-4638-b938-4285e9cce70d","Type":"ContainerStarted","Data":"e6f3440699f8dda1101bd97c0c415f035ea39c13a0c1abcf93bab7b2a60b587c"} Jul 06 21:33:29.141720 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:29.141697 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" podUID="919fd4d5-49d6-4638-b938-4285e9cce70d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Jul 06 21:33:29.160215 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:29.160167 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" podStartSLOduration=2.160150117 podStartE2EDuration="2.160150117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:33:29.157851107 +0000 UTC m=+1233.920764112" watchObservedRunningTime="2026-07-06 21:33:29.160150117 +0000 UTC m=+1233.923063165" Jul 06 21:33:29.787386 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:29.787325 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" podUID="412725d6-83f6-4641-9885-765577a11eaf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused" Jul 06 21:33:30.144822 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:30.144786 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" podUID="919fd4d5-49d6-4638-b938-4285e9cce70d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Jul 06 21:33:31.164085 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.164054 2581 generic.go:356] "Generic (PLEG): container finished" podID="412725d6-83f6-4641-9885-765577a11eaf" containerID="7795dc6430bc773a4d8bb995d5cd73ccb0c685cbc50b2e3b8ddd77385df36bab" exitCode=0 Jul 06 21:33:31.164545 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.164127 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" event={"ID":"412725d6-83f6-4641-9885-765577a11eaf","Type":"ContainerDied","Data":"7795dc6430bc773a4d8bb995d5cd73ccb0c685cbc50b2e3b8ddd77385df36bab"} Jul 06 21:33:31.166015 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.165979 2581 generic.go:356] "Generic (PLEG): container finished" podID="fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc" containerID="32e3ff29fa06739d765f1dff8f49472a5479a56fd80b69312b24f085a718dc83" exitCode=0 Jul 06 21:33:31.166118 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.166013 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" event={"ID":"fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc","Type":"ContainerDied","Data":"32e3ff29fa06739d765f1dff8f49472a5479a56fd80b69312b24f085a718dc83"} Jul 06 21:33:31.172065 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.172043 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" Jul 06 21:33:31.241510 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.241482 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" Jul 06 21:33:31.338882 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.338850 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/412725d6-83f6-4641-9885-765577a11eaf-proxy-tls\") pod \"412725d6-83f6-4641-9885-765577a11eaf\" (UID: \"412725d6-83f6-4641-9885-765577a11eaf\") " Jul 06 21:33:31.339054 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.338890 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrmsg\" (UniqueName: \"kubernetes.io/projected/412725d6-83f6-4641-9885-765577a11eaf-kube-api-access-zrmsg\") pod \"412725d6-83f6-4641-9885-765577a11eaf\" (UID: \"412725d6-83f6-4641-9885-765577a11eaf\") " Jul 06 21:33:31.339054 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.338916 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc-proxy-tls\") pod \"fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc\" (UID: \"fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc\") " Jul 06 21:33:31.339054 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.338945 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc-kube-rbac-proxy-sar-config\") pod \"fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc\" (UID: \"fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc\") " Jul 06 21:33:31.339054 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.339002 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/412725d6-83f6-4641-9885-765577a11eaf-kube-rbac-proxy-sar-config\") pod \"412725d6-83f6-4641-9885-765577a11eaf\" (UID: \"412725d6-83f6-4641-9885-765577a11eaf\") " Jul 06 21:33:31.339054 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.339034 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjp2g\" (UniqueName: \"kubernetes.io/projected/fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc-kube-api-access-xjp2g\") pod \"fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc\" (UID: \"fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc\") " Jul 06 21:33:31.339409 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.339378 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc" (UID: "fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:33:31.339493 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.339435 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/412725d6-83f6-4641-9885-765577a11eaf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "412725d6-83f6-4641-9885-765577a11eaf" (UID: "412725d6-83f6-4641-9885-765577a11eaf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:33:31.341063 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.341028 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/412725d6-83f6-4641-9885-765577a11eaf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "412725d6-83f6-4641-9885-765577a11eaf" (UID: "412725d6-83f6-4641-9885-765577a11eaf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:33:31.341063 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.341046 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc" (UID: "fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:33:31.341225 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.341136 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412725d6-83f6-4641-9885-765577a11eaf-kube-api-access-zrmsg" (OuterVolumeSpecName: "kube-api-access-zrmsg") pod "412725d6-83f6-4641-9885-765577a11eaf" (UID: "412725d6-83f6-4641-9885-765577a11eaf"). InnerVolumeSpecName "kube-api-access-zrmsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:33:31.341225 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.341157 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc-kube-api-access-xjp2g" (OuterVolumeSpecName: "kube-api-access-xjp2g") pod "fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc" (UID: "fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc"). InnerVolumeSpecName "kube-api-access-xjp2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:33:31.440237 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.440197 2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/412725d6-83f6-4641-9885-765577a11eaf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:33:31.440237 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.440228 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjp2g\" (UniqueName: \"kubernetes.io/projected/fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc-kube-api-access-xjp2g\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:33:31.440237 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.440239 2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/412725d6-83f6-4641-9885-765577a11eaf-proxy-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:33:31.440512 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.440251 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrmsg\" (UniqueName: \"kubernetes.io/projected/412725d6-83f6-4641-9885-765577a11eaf-kube-api-access-zrmsg\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:33:31.440512 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.440259 2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc-proxy-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:33:31.440512 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:31.440268 2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:33:32.172102 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:32.172072 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" Jul 06 21:33:32.172546 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:32.172072 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg" event={"ID":"412725d6-83f6-4641-9885-765577a11eaf","Type":"ContainerDied","Data":"493ee55bda3db0357fec38c7571cff44982902f464bd303847ec40ce53bd6871"} Jul 06 21:33:32.172546 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:32.172231 2581 scope.go:117] "RemoveContainer" containerID="32b87d25f8feb9e34247c10101714a484ae6fcbb2abf8ffed20a4f5b1f7c0804" Jul 06 21:33:32.173904 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:32.173883 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" Jul 06 21:33:32.174014 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:32.173905 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch" event={"ID":"fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc","Type":"ContainerDied","Data":"c7550448f13380be5eee8a66a5cc76d3bb6e227f2263c9c3495abb5f1ede44b4"} Jul 06 21:33:32.180603 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:32.180578 2581 scope.go:117] "RemoveContainer" containerID="7795dc6430bc773a4d8bb995d5cd73ccb0c685cbc50b2e3b8ddd77385df36bab" Jul 06 21:33:32.188139 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:32.188118 2581 scope.go:117] "RemoveContainer" containerID="0d805541d7de269b7a61cba6036f70f23eaa551a4e9c98e8a2b9510164e619b8" Jul 06 21:33:32.189039 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:32.188945 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg"] Jul 06 21:33:32.193412 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:32.193393 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-69e1e-predictor-68f588fff7-bq6qg"] Jul 06 21:33:32.195803 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:32.195790 2581 scope.go:117] "RemoveContainer" containerID="32e3ff29fa06739d765f1dff8f49472a5479a56fd80b69312b24f085a718dc83" Jul 06 21:33:32.209895 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:32.209872 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch"] Jul 06 21:33:32.214377 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:32.214350 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-69e1e-predictor-7568595c68-8t2ch"] Jul 06 21:33:33.786149 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:33.786117 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="412725d6-83f6-4641-9885-765577a11eaf" path="/var/lib/kubelet/pods/412725d6-83f6-4641-9885-765577a11eaf/volumes" Jul 06 21:33:33.786565 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:33.786551 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc" path="/var/lib/kubelet/pods/fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc/volumes" Jul 06 21:33:34.083620 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:34.083525 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" podUID="a4186ac0-8140-41eb-893d-b5417ec52a8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Jul 06 21:33:34.146643 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:34.146616 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" Jul 06 21:33:34.147156 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:34.147128 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" podUID="9dc8783d-9353-4694-982e-febf4d48ee85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Jul 06 21:33:35.091576 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:35.091534 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" podUID="db17289d-5948-4e9e-a0c1-481806859036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Jul 06 21:33:35.150048 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:35.150016 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" Jul 06 21:33:35.150559 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:35.150522 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" podUID="919fd4d5-49d6-4638-b938-4285e9cce70d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Jul 06 21:33:44.083725 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:44.083682 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" podUID="a4186ac0-8140-41eb-893d-b5417ec52a8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Jul 06 21:33:44.147797 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:44.147751 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" podUID="9dc8783d-9353-4694-982e-febf4d48ee85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Jul 06 21:33:45.091424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:45.091372 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" podUID="db17289d-5948-4e9e-a0c1-481806859036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Jul 06 21:33:45.151098 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:45.151053 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" podUID="919fd4d5-49d6-4638-b938-4285e9cce70d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Jul 06 21:33:54.083635 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:54.083551 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" podUID="a4186ac0-8140-41eb-893d-b5417ec52a8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Jul 06 21:33:54.148036 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:54.147974 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" podUID="9dc8783d-9353-4694-982e-febf4d48ee85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Jul 06 21:33:55.091166 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:55.091122 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" podUID="db17289d-5948-4e9e-a0c1-481806859036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Jul 06 21:33:55.150528 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:33:55.150481 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" podUID="919fd4d5-49d6-4638-b938-4285e9cce70d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Jul 06 21:34:04.083874 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:04.083831 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" podUID="a4186ac0-8140-41eb-893d-b5417ec52a8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Jul 06 21:34:04.147919 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:04.147871 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" podUID="9dc8783d-9353-4694-982e-febf4d48ee85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Jul 06 21:34:05.091386 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:05.091325 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" podUID="db17289d-5948-4e9e-a0c1-481806859036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Jul 06 21:34:05.151536 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:05.151484 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" podUID="919fd4d5-49d6-4638-b938-4285e9cce70d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Jul 06 21:34:14.085844 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:14.085807 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" Jul 06 21:34:14.147654 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:14.147617 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" podUID="9dc8783d-9353-4694-982e-febf4d48ee85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Jul 06 21:34:15.091486 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:15.091460 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" Jul 06 21:34:15.150746 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:15.150706 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" podUID="919fd4d5-49d6-4638-b938-4285e9cce70d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Jul 06 21:34:24.148144 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:24.148103 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" Jul 06 21:34:25.151902 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:25.151872 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" Jul 06 21:34:37.016877 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.016836 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6"] Jul 06 21:34:37.017381 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.017131 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" podUID="db17289d-5948-4e9e-a0c1-481806859036" containerName="kserve-container" containerID="cri-o://047453869c6ef0f79b43c22347d236dc166ce8dd9f81498e8baa6b85a85b168b" gracePeriod=30 Jul 06 21:34:37.017381 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.017172 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" podUID="db17289d-5948-4e9e-a0c1-481806859036" containerName="kube-rbac-proxy" containerID="cri-o://b4311b586c617e397650e0e76c45ddbfa87de1114396b20d6a1505d49379aa86" gracePeriod=30 Jul 06 21:34:37.032173 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.032149 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb"] Jul 06 21:34:37.032543 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.032531 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc" containerName="kserve-container" Jul 06 21:34:37.032589 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.032546 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc" containerName="kserve-container" Jul 06 21:34:37.032589 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.032555 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="412725d6-83f6-4641-9885-765577a11eaf" containerName="kserve-container" Jul 06 21:34:37.032589 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.032560 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="412725d6-83f6-4641-9885-765577a11eaf" containerName="kserve-container" Jul 06 21:34:37.032589 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.032570 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="412725d6-83f6-4641-9885-765577a11eaf" containerName="kube-rbac-proxy" Jul 06 21:34:37.032589 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.032577 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="412725d6-83f6-4641-9885-765577a11eaf" containerName="kube-rbac-proxy" Jul 06 21:34:37.032747 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.032591 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc" containerName="kube-rbac-proxy" Jul 06 21:34:37.032747 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.032596 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc" containerName="kube-rbac-proxy" Jul 06 21:34:37.032747 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.032650 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="412725d6-83f6-4641-9885-765577a11eaf" containerName="kube-rbac-proxy" Jul 06 21:34:37.032747 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.032658 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc" containerName="kube-rbac-proxy" Jul 06 21:34:37.032747 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.032666 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="412725d6-83f6-4641-9885-765577a11eaf" containerName="kserve-container" Jul 06 21:34:37.032747 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.032673 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdee1a4b-bae2-4f37-8fb9-3acc1f8c64cc" containerName="kserve-container" Jul 06 21:34:37.035980 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.035962 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" Jul 06 21:34:37.038522 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.038500 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a54dc-predictor-serving-cert\"" Jul 06 21:34:37.038628 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.038588 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a54dc-kube-rbac-proxy-sar-config\"" Jul 06 21:34:37.048306 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.048281 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb"] Jul 06 21:34:37.102040 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.102011 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/faad73e2-ca54-4e10-8af0-003bbc901de6-proxy-tls\") pod \"success-200-isvc-a54dc-predictor-558db86446-4lqqb\" (UID: \"faad73e2-ca54-4e10-8af0-003bbc901de6\") " pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" Jul 06 21:34:37.102206 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.102046 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vst4w\" (UniqueName: \"kubernetes.io/projected/faad73e2-ca54-4e10-8af0-003bbc901de6-kube-api-access-vst4w\") pod \"success-200-isvc-a54dc-predictor-558db86446-4lqqb\" (UID: \"faad73e2-ca54-4e10-8af0-003bbc901de6\") " pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" Jul 06 21:34:37.102206 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.102078 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/faad73e2-ca54-4e10-8af0-003bbc901de6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a54dc-predictor-558db86446-4lqqb\" (UID: \"faad73e2-ca54-4e10-8af0-003bbc901de6\") " pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" Jul 06 21:34:37.142130 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.142101 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh"] Jul 06 21:34:37.147927 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.147893 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" Jul 06 21:34:37.150553 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.150529 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a54dc-predictor-serving-cert\"" Jul 06 21:34:37.150680 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.150604 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a54dc-kube-rbac-proxy-sar-config\"" Jul 06 21:34:37.156783 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.156757 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh"] Jul 06 21:34:37.202755 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.202720 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7830a974-68bd-462d-addc-faad0a1735d5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh\" (UID: \"7830a974-68bd-462d-addc-faad0a1735d5\") " pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" Jul 06 21:34:37.202903 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.202762 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7830a974-68bd-462d-addc-faad0a1735d5-proxy-tls\") pod \"error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh\" (UID: \"7830a974-68bd-462d-addc-faad0a1735d5\") " pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" Jul 06 21:34:37.202903 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.202783 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckf8j\" (UniqueName: \"kubernetes.io/projected/7830a974-68bd-462d-addc-faad0a1735d5-kube-api-access-ckf8j\") pod \"error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh\" (UID: \"7830a974-68bd-462d-addc-faad0a1735d5\") " pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" Jul 06 21:34:37.202903 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.202886 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/faad73e2-ca54-4e10-8af0-003bbc901de6-proxy-tls\") pod \"success-200-isvc-a54dc-predictor-558db86446-4lqqb\" (UID: \"faad73e2-ca54-4e10-8af0-003bbc901de6\") " pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" Jul 06 21:34:37.203036 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.202930 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vst4w\" (UniqueName: \"kubernetes.io/projected/faad73e2-ca54-4e10-8af0-003bbc901de6-kube-api-access-vst4w\") pod \"success-200-isvc-a54dc-predictor-558db86446-4lqqb\" (UID: \"faad73e2-ca54-4e10-8af0-003bbc901de6\") " pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" Jul 06 21:34:37.203036 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.202970 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/faad73e2-ca54-4e10-8af0-003bbc901de6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a54dc-predictor-558db86446-4lqqb\" (UID: \"faad73e2-ca54-4e10-8af0-003bbc901de6\") " pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" Jul 06 21:34:37.203765 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.203742 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/faad73e2-ca54-4e10-8af0-003bbc901de6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a54dc-predictor-558db86446-4lqqb\" (UID: \"faad73e2-ca54-4e10-8af0-003bbc901de6\") " pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" Jul 06 21:34:37.205508 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.205487 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/faad73e2-ca54-4e10-8af0-003bbc901de6-proxy-tls\") pod \"success-200-isvc-a54dc-predictor-558db86446-4lqqb\" (UID: \"faad73e2-ca54-4e10-8af0-003bbc901de6\") " pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" Jul 06 21:34:37.222702 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.222675 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vst4w\" (UniqueName: \"kubernetes.io/projected/faad73e2-ca54-4e10-8af0-003bbc901de6-kube-api-access-vst4w\") pod \"success-200-isvc-a54dc-predictor-558db86446-4lqqb\" (UID: \"faad73e2-ca54-4e10-8af0-003bbc901de6\") " pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" Jul 06 21:34:37.235500 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.235477 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst"] Jul 06 21:34:37.235714 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.235693 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" podUID="a4186ac0-8140-41eb-893d-b5417ec52a8f" containerName="kserve-container" containerID="cri-o://9d5bc6c4f56f280c56d5a1b09d267259f6e852596074dac3230dee584b735212" gracePeriod=30 Jul 06 21:34:37.235795 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.235721 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" podUID="a4186ac0-8140-41eb-893d-b5417ec52a8f" containerName="kube-rbac-proxy" containerID="cri-o://b42620d28f89b5820ac56099bfa14a279dd9d893d9e32a346dcf3eaaf9634a70" gracePeriod=30 Jul 06 21:34:37.303953 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.303882 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7830a974-68bd-462d-addc-faad0a1735d5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh\" (UID: \"7830a974-68bd-462d-addc-faad0a1735d5\") " pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" Jul 06 21:34:37.303953 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.303919 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7830a974-68bd-462d-addc-faad0a1735d5-proxy-tls\") pod \"error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh\" (UID: \"7830a974-68bd-462d-addc-faad0a1735d5\") " pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" Jul 06 21:34:37.303953 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.303938 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckf8j\" (UniqueName: \"kubernetes.io/projected/7830a974-68bd-462d-addc-faad0a1735d5-kube-api-access-ckf8j\") pod \"error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh\" (UID: \"7830a974-68bd-462d-addc-faad0a1735d5\") " pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" Jul 06 21:34:37.304536 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.304512 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7830a974-68bd-462d-addc-faad0a1735d5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh\" (UID: \"7830a974-68bd-462d-addc-faad0a1735d5\") " pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" Jul 06 21:34:37.306374 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.306330 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7830a974-68bd-462d-addc-faad0a1735d5-proxy-tls\") pod \"error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh\" (UID: \"7830a974-68bd-462d-addc-faad0a1735d5\") " pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" Jul 06 21:34:37.312231 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.312207 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckf8j\" (UniqueName: \"kubernetes.io/projected/7830a974-68bd-462d-addc-faad0a1735d5-kube-api-access-ckf8j\") pod \"error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh\" (UID: \"7830a974-68bd-462d-addc-faad0a1735d5\") " pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" Jul 06 21:34:37.346351 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.346306 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" Jul 06 21:34:37.460932 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.460740 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" Jul 06 21:34:37.462732 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.462706 2581 generic.go:356] "Generic (PLEG): container finished" podID="db17289d-5948-4e9e-a0c1-481806859036" containerID="b4311b586c617e397650e0e76c45ddbfa87de1114396b20d6a1505d49379aa86" exitCode=2 Jul 06 21:34:37.462865 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.462784 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" event={"ID":"db17289d-5948-4e9e-a0c1-481806859036","Type":"ContainerDied","Data":"b4311b586c617e397650e0e76c45ddbfa87de1114396b20d6a1505d49379aa86"} Jul 06 21:34:37.464688 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.464651 2581 generic.go:356] "Generic (PLEG): container finished" podID="a4186ac0-8140-41eb-893d-b5417ec52a8f" containerID="b42620d28f89b5820ac56099bfa14a279dd9d893d9e32a346dcf3eaaf9634a70" exitCode=2 Jul 06 21:34:37.464742 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.464680 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" event={"ID":"a4186ac0-8140-41eb-893d-b5417ec52a8f","Type":"ContainerDied","Data":"b42620d28f89b5820ac56099bfa14a279dd9d893d9e32a346dcf3eaaf9634a70"} Jul 06 21:34:37.483412 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.483389 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb"] Jul 06 21:34:37.485892 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:34:37.485870 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaad73e2_ca54_4e10_8af0_003bbc901de6.slice/crio-723d80c005bdc364755f36404a05b4f837360eee08f38505a4eb9a9231dfc77e WatchSource:0}: Error finding container 723d80c005bdc364755f36404a05b4f837360eee08f38505a4eb9a9231dfc77e: Status 404 returned error can't find the container with id 723d80c005bdc364755f36404a05b4f837360eee08f38505a4eb9a9231dfc77e Jul 06 21:34:37.600231 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:37.600159 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh"] Jul 06 21:34:37.602046 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:34:37.602016 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7830a974_68bd_462d_addc_faad0a1735d5.slice/crio-e8955f32c6047075283a37a59639435765eddc24fa5a91758375ce6e730576c3 WatchSource:0}: Error finding container e8955f32c6047075283a37a59639435765eddc24fa5a91758375ce6e730576c3: Status 404 returned error can't find the container with id e8955f32c6047075283a37a59639435765eddc24fa5a91758375ce6e730576c3 Jul 06 21:34:38.471496 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:38.471452 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" event={"ID":"7830a974-68bd-462d-addc-faad0a1735d5","Type":"ContainerStarted","Data":"188a4de7e53802852ea047fbdc2872e05636e9f36f34fcb6a54232539f7240c8"} Jul 06 21:34:38.471969 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:38.471615 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" Jul 06 21:34:38.471969 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:38.471678 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" event={"ID":"7830a974-68bd-462d-addc-faad0a1735d5","Type":"ContainerStarted","Data":"07c882e08eab3e813db106965f38729872fcc4ffa3e8788d8aa726a85232f44f"} Jul 06 21:34:38.471969 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:38.471715 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" Jul 06 21:34:38.471969 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:38.471745 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" event={"ID":"7830a974-68bd-462d-addc-faad0a1735d5","Type":"ContainerStarted","Data":"e8955f32c6047075283a37a59639435765eddc24fa5a91758375ce6e730576c3"} Jul 06 21:34:38.472728 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:38.472704 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" podUID="7830a974-68bd-462d-addc-faad0a1735d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Jul 06 21:34:38.473387 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:38.473357 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" event={"ID":"faad73e2-ca54-4e10-8af0-003bbc901de6","Type":"ContainerStarted","Data":"b836709a3a02c823e5354a0c729d2b2e53c6a79015a1c596f901719aa60b7255"} Jul 06 21:34:38.473497 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:38.473398 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" event={"ID":"faad73e2-ca54-4e10-8af0-003bbc901de6","Type":"ContainerStarted","Data":"4dcc035d10208605a3ffa3ac1af744e8e4518352ee0f943b5bdad1a6d85ee4e5"} Jul 06 21:34:38.473497 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:38.473418 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" event={"ID":"faad73e2-ca54-4e10-8af0-003bbc901de6","Type":"ContainerStarted","Data":"723d80c005bdc364755f36404a05b4f837360eee08f38505a4eb9a9231dfc77e"} Jul 06 21:34:38.473497 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:38.473452 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" Jul 06 21:34:38.492332 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:38.492285 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" podStartSLOduration=1.492273217 podStartE2EDuration="1.492273217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:34:38.490311661 +0000 UTC m=+1303.253224665" watchObservedRunningTime="2026-07-06 21:34:38.492273217 +0000 UTC m=+1303.255186220" Jul 06 21:34:38.513583 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:38.513517 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" podStartSLOduration=1.5134991690000001 podStartE2EDuration="1.513499169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:34:38.512677113 +0000 UTC m=+1303.275590117" watchObservedRunningTime="2026-07-06 21:34:38.513499169 +0000 UTC m=+1303.276412174" Jul 06 21:34:39.079242 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:39.079196 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" podUID="a4186ac0-8140-41eb-893d-b5417ec52a8f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused" Jul 06 21:34:39.478133 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:39.478086 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" podUID="7830a974-68bd-462d-addc-faad0a1735d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Jul 06 21:34:39.478623 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:39.478305 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" Jul 06 21:34:39.479489 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:39.479465 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" podUID="faad73e2-ca54-4e10-8af0-003bbc901de6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 06 21:34:40.086719 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.086685 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" podUID="db17289d-5948-4e9e-a0c1-481806859036" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused" Jul 06 21:34:40.371879 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.371856 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" Jul 06 21:34:40.427547 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.427496 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db17289d-5948-4e9e-a0c1-481806859036-proxy-tls\") pod \"db17289d-5948-4e9e-a0c1-481806859036\" (UID: \"db17289d-5948-4e9e-a0c1-481806859036\") " Jul 06 21:34:40.427692 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.427649 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db17289d-5948-4e9e-a0c1-481806859036-kube-rbac-proxy-sar-config\") pod \"db17289d-5948-4e9e-a0c1-481806859036\" (UID: \"db17289d-5948-4e9e-a0c1-481806859036\") " Jul 06 21:34:40.427773 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.427687 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gx4w\" (UniqueName: \"kubernetes.io/projected/db17289d-5948-4e9e-a0c1-481806859036-kube-api-access-5gx4w\") pod \"db17289d-5948-4e9e-a0c1-481806859036\" (UID: \"db17289d-5948-4e9e-a0c1-481806859036\") " Jul 06 21:34:40.428095 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.428066 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db17289d-5948-4e9e-a0c1-481806859036-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "db17289d-5948-4e9e-a0c1-481806859036" (UID: "db17289d-5948-4e9e-a0c1-481806859036"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:34:40.429766 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.429743 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db17289d-5948-4e9e-a0c1-481806859036-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "db17289d-5948-4e9e-a0c1-481806859036" (UID: "db17289d-5948-4e9e-a0c1-481806859036"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:34:40.429857 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.429837 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db17289d-5948-4e9e-a0c1-481806859036-kube-api-access-5gx4w" (OuterVolumeSpecName: "kube-api-access-5gx4w") pod "db17289d-5948-4e9e-a0c1-481806859036" (UID: "db17289d-5948-4e9e-a0c1-481806859036"). InnerVolumeSpecName "kube-api-access-5gx4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:34:40.483675 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.483645 2581 generic.go:356] "Generic (PLEG): container finished" podID="db17289d-5948-4e9e-a0c1-481806859036" containerID="047453869c6ef0f79b43c22347d236dc166ce8dd9f81498e8baa6b85a85b168b" exitCode=0 Jul 06 21:34:40.484001 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.483703 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" Jul 06 21:34:40.484001 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.483728 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" event={"ID":"db17289d-5948-4e9e-a0c1-481806859036","Type":"ContainerDied","Data":"047453869c6ef0f79b43c22347d236dc166ce8dd9f81498e8baa6b85a85b168b"} Jul 06 21:34:40.484001 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.483819 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6" event={"ID":"db17289d-5948-4e9e-a0c1-481806859036","Type":"ContainerDied","Data":"d4d60098fa62e9a35d9753c61b89948c856d5a601a8c5930b4f91b9bcd6eda55"} Jul 06 21:34:40.484001 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.483849 2581 scope.go:117] "RemoveContainer" containerID="b4311b586c617e397650e0e76c45ddbfa87de1114396b20d6a1505d49379aa86" Jul 06 21:34:40.484312 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.484286 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" podUID="faad73e2-ca54-4e10-8af0-003bbc901de6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 06 21:34:40.492587 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.492567 2581 scope.go:117] "RemoveContainer" containerID="047453869c6ef0f79b43c22347d236dc166ce8dd9f81498e8baa6b85a85b168b" Jul 06 21:34:40.500225 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.500207 2581 scope.go:117] "RemoveContainer" containerID="b4311b586c617e397650e0e76c45ddbfa87de1114396b20d6a1505d49379aa86" Jul 06 21:34:40.500508 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:34:40.500485 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4311b586c617e397650e0e76c45ddbfa87de1114396b20d6a1505d49379aa86\": container with ID starting with b4311b586c617e397650e0e76c45ddbfa87de1114396b20d6a1505d49379aa86 not found: ID does not exist" containerID="b4311b586c617e397650e0e76c45ddbfa87de1114396b20d6a1505d49379aa86" Jul 06 21:34:40.500569 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.500518 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4311b586c617e397650e0e76c45ddbfa87de1114396b20d6a1505d49379aa86"} err="failed to get container status \"b4311b586c617e397650e0e76c45ddbfa87de1114396b20d6a1505d49379aa86\": rpc error: code = NotFound desc = could not find container \"b4311b586c617e397650e0e76c45ddbfa87de1114396b20d6a1505d49379aa86\": container with ID starting with b4311b586c617e397650e0e76c45ddbfa87de1114396b20d6a1505d49379aa86 not found: ID does not exist" Jul 06 21:34:40.500569 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.500541 2581 scope.go:117] "RemoveContainer" containerID="047453869c6ef0f79b43c22347d236dc166ce8dd9f81498e8baa6b85a85b168b" Jul 06 21:34:40.500785 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:34:40.500768 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"047453869c6ef0f79b43c22347d236dc166ce8dd9f81498e8baa6b85a85b168b\": container with ID starting with 047453869c6ef0f79b43c22347d236dc166ce8dd9f81498e8baa6b85a85b168b not found: ID does not exist" containerID="047453869c6ef0f79b43c22347d236dc166ce8dd9f81498e8baa6b85a85b168b" Jul 06 21:34:40.500829 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.500792 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"047453869c6ef0f79b43c22347d236dc166ce8dd9f81498e8baa6b85a85b168b"} err="failed to get container status \"047453869c6ef0f79b43c22347d236dc166ce8dd9f81498e8baa6b85a85b168b\": rpc error: code = NotFound desc = could not find container \"047453869c6ef0f79b43c22347d236dc166ce8dd9f81498e8baa6b85a85b168b\": container with ID starting with 047453869c6ef0f79b43c22347d236dc166ce8dd9f81498e8baa6b85a85b168b not found: ID does not exist" Jul 06 21:34:40.513945 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.513923 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6"] Jul 06 21:34:40.524590 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.524559 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-599ee-predictor-58f56bd85d-96fm6"] Jul 06 21:34:40.528478 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.528463 2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db17289d-5948-4e9e-a0c1-481806859036-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:34:40.528561 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.528480 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gx4w\" (UniqueName: \"kubernetes.io/projected/db17289d-5948-4e9e-a0c1-481806859036-kube-api-access-5gx4w\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:34:40.528561 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:40.528494 2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db17289d-5948-4e9e-a0c1-481806859036-proxy-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:34:41.072665 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.072635 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" Jul 06 21:34:41.133147 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.133072 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84sfd\" (UniqueName: \"kubernetes.io/projected/a4186ac0-8140-41eb-893d-b5417ec52a8f-kube-api-access-84sfd\") pod \"a4186ac0-8140-41eb-893d-b5417ec52a8f\" (UID: \"a4186ac0-8140-41eb-893d-b5417ec52a8f\") " Jul 06 21:34:41.133147 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.133118 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4186ac0-8140-41eb-893d-b5417ec52a8f-proxy-tls\") pod \"a4186ac0-8140-41eb-893d-b5417ec52a8f\" (UID: \"a4186ac0-8140-41eb-893d-b5417ec52a8f\") " Jul 06 21:34:41.133319 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.133187 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4186ac0-8140-41eb-893d-b5417ec52a8f-kube-rbac-proxy-sar-config\") pod \"a4186ac0-8140-41eb-893d-b5417ec52a8f\" (UID: \"a4186ac0-8140-41eb-893d-b5417ec52a8f\") " Jul 06 21:34:41.133589 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.133564 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4186ac0-8140-41eb-893d-b5417ec52a8f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a4186ac0-8140-41eb-893d-b5417ec52a8f" (UID: "a4186ac0-8140-41eb-893d-b5417ec52a8f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:34:41.135017 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.134998 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4186ac0-8140-41eb-893d-b5417ec52a8f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a4186ac0-8140-41eb-893d-b5417ec52a8f" (UID: "a4186ac0-8140-41eb-893d-b5417ec52a8f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:34:41.135065 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.135011 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4186ac0-8140-41eb-893d-b5417ec52a8f-kube-api-access-84sfd" (OuterVolumeSpecName: "kube-api-access-84sfd") pod "a4186ac0-8140-41eb-893d-b5417ec52a8f" (UID: "a4186ac0-8140-41eb-893d-b5417ec52a8f"). InnerVolumeSpecName "kube-api-access-84sfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:34:41.234084 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.234053 2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4186ac0-8140-41eb-893d-b5417ec52a8f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:34:41.234084 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.234079 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84sfd\" (UniqueName: \"kubernetes.io/projected/a4186ac0-8140-41eb-893d-b5417ec52a8f-kube-api-access-84sfd\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:34:41.234224 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.234090 2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4186ac0-8140-41eb-893d-b5417ec52a8f-proxy-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:34:41.491607 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.491523 2581 generic.go:356] "Generic (PLEG): container finished" podID="a4186ac0-8140-41eb-893d-b5417ec52a8f" containerID="9d5bc6c4f56f280c56d5a1b09d267259f6e852596074dac3230dee584b735212" exitCode=0 Jul 06 21:34:41.491607 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.491575 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" Jul 06 21:34:41.491607 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.491592 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" event={"ID":"a4186ac0-8140-41eb-893d-b5417ec52a8f","Type":"ContainerDied","Data":"9d5bc6c4f56f280c56d5a1b09d267259f6e852596074dac3230dee584b735212"} Jul 06 21:34:41.492151 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.491638 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst" event={"ID":"a4186ac0-8140-41eb-893d-b5417ec52a8f","Type":"ContainerDied","Data":"99d133df8e278bc9e7f400e142fc14305cd9d908ecec487bc12600033991cc63"} Jul 06 21:34:41.492151 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.491665 2581 scope.go:117] "RemoveContainer" containerID="b42620d28f89b5820ac56099bfa14a279dd9d893d9e32a346dcf3eaaf9634a70" Jul 06 21:34:41.500584 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.500568 2581 scope.go:117] "RemoveContainer" containerID="9d5bc6c4f56f280c56d5a1b09d267259f6e852596074dac3230dee584b735212" Jul 06 21:34:41.509808 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.509780 2581 scope.go:117] "RemoveContainer" containerID="b42620d28f89b5820ac56099bfa14a279dd9d893d9e32a346dcf3eaaf9634a70" Jul 06 21:34:41.510264 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:34:41.510244 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b42620d28f89b5820ac56099bfa14a279dd9d893d9e32a346dcf3eaaf9634a70\": container with ID starting with b42620d28f89b5820ac56099bfa14a279dd9d893d9e32a346dcf3eaaf9634a70 not found: ID does not exist" containerID="b42620d28f89b5820ac56099bfa14a279dd9d893d9e32a346dcf3eaaf9634a70" Jul 06 21:34:41.510313 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.510272 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b42620d28f89b5820ac56099bfa14a279dd9d893d9e32a346dcf3eaaf9634a70"} err="failed to get container status \"b42620d28f89b5820ac56099bfa14a279dd9d893d9e32a346dcf3eaaf9634a70\": rpc error: code = NotFound desc = could not find container \"b42620d28f89b5820ac56099bfa14a279dd9d893d9e32a346dcf3eaaf9634a70\": container with ID starting with b42620d28f89b5820ac56099bfa14a279dd9d893d9e32a346dcf3eaaf9634a70 not found: ID does not exist" Jul 06 21:34:41.510313 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.510288 2581 scope.go:117] "RemoveContainer" containerID="9d5bc6c4f56f280c56d5a1b09d267259f6e852596074dac3230dee584b735212" Jul 06 21:34:41.510565 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:34:41.510545 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5bc6c4f56f280c56d5a1b09d267259f6e852596074dac3230dee584b735212\": container with ID starting with 9d5bc6c4f56f280c56d5a1b09d267259f6e852596074dac3230dee584b735212 not found: ID does not exist" containerID="9d5bc6c4f56f280c56d5a1b09d267259f6e852596074dac3230dee584b735212" Jul 06 21:34:41.510647 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.510569 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5bc6c4f56f280c56d5a1b09d267259f6e852596074dac3230dee584b735212"} err="failed to get container status \"9d5bc6c4f56f280c56d5a1b09d267259f6e852596074dac3230dee584b735212\": rpc error: code = NotFound desc = could not find container \"9d5bc6c4f56f280c56d5a1b09d267259f6e852596074dac3230dee584b735212\": container with ID starting with 9d5bc6c4f56f280c56d5a1b09d267259f6e852596074dac3230dee584b735212 not found: ID does not exist" Jul 06 21:34:41.513829 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.513809 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst"] Jul 06 21:34:41.518008 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.517987 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-599ee-predictor-6fdd64d5bc-nplst"] Jul 06 21:34:41.786706 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.786627 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4186ac0-8140-41eb-893d-b5417ec52a8f" path="/var/lib/kubelet/pods/a4186ac0-8140-41eb-893d-b5417ec52a8f/volumes" Jul 06 21:34:41.787042 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:41.787029 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db17289d-5948-4e9e-a0c1-481806859036" path="/var/lib/kubelet/pods/db17289d-5948-4e9e-a0c1-481806859036/volumes" Jul 06 21:34:44.482218 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:44.482190 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" Jul 06 21:34:44.482682 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:44.482657 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" podUID="7830a974-68bd-462d-addc-faad0a1735d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Jul 06 21:34:45.489402 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:45.489371 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" Jul 06 21:34:45.489830 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:45.489758 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" podUID="faad73e2-ca54-4e10-8af0-003bbc901de6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 06 21:34:54.483034 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:54.482994 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" podUID="7830a974-68bd-462d-addc-faad0a1735d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Jul 06 21:34:55.489864 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:34:55.489821 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" podUID="faad73e2-ca54-4e10-8af0-003bbc901de6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 06 21:35:04.483412 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:35:04.483367 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" podUID="7830a974-68bd-462d-addc-faad0a1735d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Jul 06 21:35:05.489938 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:35:05.489894 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" podUID="faad73e2-ca54-4e10-8af0-003bbc901de6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 06 21:35:14.483605 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:35:14.483560 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" podUID="7830a974-68bd-462d-addc-faad0a1735d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Jul 06 21:35:15.490097 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:35:15.490056 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" podUID="faad73e2-ca54-4e10-8af0-003bbc901de6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 06 21:35:24.484131 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:35:24.484044 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" Jul 06 21:35:25.491025 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:35:25.490996 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" Jul 06 21:42:52.051961 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.051886 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd"] Jul 06 21:42:52.054423 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.052505 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4186ac0-8140-41eb-893d-b5417ec52a8f" containerName="kserve-container" Jul 06 21:42:52.054423 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.052522 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4186ac0-8140-41eb-893d-b5417ec52a8f" containerName="kserve-container" Jul 06 21:42:52.054423 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.052540 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db17289d-5948-4e9e-a0c1-481806859036" containerName="kube-rbac-proxy" Jul 06 21:42:52.054423 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.052546 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="db17289d-5948-4e9e-a0c1-481806859036" containerName="kube-rbac-proxy" Jul 06 21:42:52.054423 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.052557 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4186ac0-8140-41eb-893d-b5417ec52a8f" containerName="kube-rbac-proxy" Jul 06 21:42:52.054423 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.052563 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4186ac0-8140-41eb-893d-b5417ec52a8f" containerName="kube-rbac-proxy" Jul 06 21:42:52.054423 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.052572 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db17289d-5948-4e9e-a0c1-481806859036" containerName="kserve-container" Jul 06 21:42:52.054423 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.052577 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="db17289d-5948-4e9e-a0c1-481806859036" containerName="kserve-container" Jul 06 21:42:52.054423 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.052665 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="db17289d-5948-4e9e-a0c1-481806859036" containerName="kserve-container" Jul 06 21:42:52.054423 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.052675 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="db17289d-5948-4e9e-a0c1-481806859036" containerName="kube-rbac-proxy" Jul 06 21:42:52.054423 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.052685 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4186ac0-8140-41eb-893d-b5417ec52a8f" containerName="kserve-container" Jul 06 21:42:52.054423 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.052692 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4186ac0-8140-41eb-893d-b5417ec52a8f" containerName="kube-rbac-proxy" Jul 06 21:42:52.055602 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.055581 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" Jul 06 21:42:52.058443 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.058420 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-974ed-kube-rbac-proxy-sar-config\"" Jul 06 21:42:52.058571 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.058451 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-974ed-predictor-serving-cert\"" Jul 06 21:42:52.064312 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.064286 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp"] Jul 06 21:42:52.064568 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.064528 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" podUID="9dc8783d-9353-4694-982e-febf4d48ee85" containerName="kserve-container" containerID="cri-o://dc5d88742458cb07440b471f2241d7619fac6180864125e9938dc6da5b805497" gracePeriod=30 Jul 06 21:42:52.064645 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.064607 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" podUID="9dc8783d-9353-4694-982e-febf4d48ee85" containerName="kube-rbac-proxy" containerID="cri-o://9720f2a5dd1ab6342980ae39a333ed875f3c2032199c73d2d06a8b7edd4fc0e3" gracePeriod=30 Jul 06 21:42:52.068820 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.068751 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd"] Jul 06 21:42:52.115227 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.115193 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f0d4bf0-2bec-413c-98d1-e3a5f264ab26-proxy-tls\") pod \"success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd\" (UID: \"1f0d4bf0-2bec-413c-98d1-e3a5f264ab26\") " pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" Jul 06 21:42:52.115424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.115258 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r64vg\" (UniqueName: \"kubernetes.io/projected/1f0d4bf0-2bec-413c-98d1-e3a5f264ab26-kube-api-access-r64vg\") pod \"success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd\" (UID: \"1f0d4bf0-2bec-413c-98d1-e3a5f264ab26\") " pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" Jul 06 21:42:52.115424 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.115301 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f0d4bf0-2bec-413c-98d1-e3a5f264ab26-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd\" (UID: \"1f0d4bf0-2bec-413c-98d1-e3a5f264ab26\") " pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" Jul 06 21:42:52.126488 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.126460 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8"] Jul 06 21:42:52.129975 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.129959 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" Jul 06 21:42:52.132297 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.132273 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-974ed-predictor-serving-cert\"" Jul 06 21:42:52.132410 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.132320 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-974ed-kube-rbac-proxy-sar-config\"" Jul 06 21:42:52.139478 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.139451 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8"] Jul 06 21:42:52.173283 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.173254 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx"] Jul 06 21:42:52.173519 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.173493 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" podUID="919fd4d5-49d6-4638-b938-4285e9cce70d" containerName="kserve-container" containerID="cri-o://c90c86031c7aefac581108b58bdffd1900ff461e47acee7d74b4fdbfa6e3b696" gracePeriod=30 Jul 06 21:42:52.173602 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.173545 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" podUID="919fd4d5-49d6-4638-b938-4285e9cce70d" containerName="kube-rbac-proxy" containerID="cri-o://a3038ba06b576c81bcb8ec3d07c41490561cf0ae2bd12adce659f1481e04fd4e" gracePeriod=30 Jul 06 21:42:52.215771 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.215740 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/639574ec-8467-4bb3-9203-6168444f8331-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-974ed-predictor-677df94fc9-c69t8\" (UID: \"639574ec-8467-4bb3-9203-6168444f8331\") " pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" Jul 06 21:42:52.215939 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.215783 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f0d4bf0-2bec-413c-98d1-e3a5f264ab26-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd\" (UID: \"1f0d4bf0-2bec-413c-98d1-e3a5f264ab26\") " pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" Jul 06 21:42:52.215939 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.215867 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f0d4bf0-2bec-413c-98d1-e3a5f264ab26-proxy-tls\") pod \"success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd\" (UID: \"1f0d4bf0-2bec-413c-98d1-e3a5f264ab26\") " pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" Jul 06 21:42:52.215939 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.215924 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6jnl\" (UniqueName: \"kubernetes.io/projected/639574ec-8467-4bb3-9203-6168444f8331-kube-api-access-m6jnl\") pod \"error-404-isvc-974ed-predictor-677df94fc9-c69t8\" (UID: \"639574ec-8467-4bb3-9203-6168444f8331\") " pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" Jul 06 21:42:52.216057 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.215952 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/639574ec-8467-4bb3-9203-6168444f8331-proxy-tls\") pod \"error-404-isvc-974ed-predictor-677df94fc9-c69t8\" (UID: \"639574ec-8467-4bb3-9203-6168444f8331\") " pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" Jul 06 21:42:52.216057 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.216021 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r64vg\" (UniqueName: \"kubernetes.io/projected/1f0d4bf0-2bec-413c-98d1-e3a5f264ab26-kube-api-access-r64vg\") pod \"success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd\" (UID: \"1f0d4bf0-2bec-413c-98d1-e3a5f264ab26\") " pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" Jul 06 21:42:52.216384 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.216367 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f0d4bf0-2bec-413c-98d1-e3a5f264ab26-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd\" (UID: \"1f0d4bf0-2bec-413c-98d1-e3a5f264ab26\") " pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" Jul 06 21:42:52.218329 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.218308 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f0d4bf0-2bec-413c-98d1-e3a5f264ab26-proxy-tls\") pod \"success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd\" (UID: \"1f0d4bf0-2bec-413c-98d1-e3a5f264ab26\") " pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" Jul 06 21:42:52.227164 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.227142 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r64vg\" (UniqueName: \"kubernetes.io/projected/1f0d4bf0-2bec-413c-98d1-e3a5f264ab26-kube-api-access-r64vg\") pod \"success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd\" (UID: \"1f0d4bf0-2bec-413c-98d1-e3a5f264ab26\") " pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" Jul 06 21:42:52.316805 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.316726 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6jnl\" (UniqueName: \"kubernetes.io/projected/639574ec-8467-4bb3-9203-6168444f8331-kube-api-access-m6jnl\") pod \"error-404-isvc-974ed-predictor-677df94fc9-c69t8\" (UID: \"639574ec-8467-4bb3-9203-6168444f8331\") " pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" Jul 06 21:42:52.316805 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.316761 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/639574ec-8467-4bb3-9203-6168444f8331-proxy-tls\") pod \"error-404-isvc-974ed-predictor-677df94fc9-c69t8\" (UID: \"639574ec-8467-4bb3-9203-6168444f8331\") " pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" Jul 06 21:42:52.316805 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.316795 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/639574ec-8467-4bb3-9203-6168444f8331-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-974ed-predictor-677df94fc9-c69t8\" (UID: \"639574ec-8467-4bb3-9203-6168444f8331\") " pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" Jul 06 21:42:52.317527 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.317502 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/639574ec-8467-4bb3-9203-6168444f8331-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-974ed-predictor-677df94fc9-c69t8\" (UID: \"639574ec-8467-4bb3-9203-6168444f8331\") " pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" Jul 06 21:42:52.319215 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.319196 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/639574ec-8467-4bb3-9203-6168444f8331-proxy-tls\") pod \"error-404-isvc-974ed-predictor-677df94fc9-c69t8\" (UID: \"639574ec-8467-4bb3-9203-6168444f8331\") " pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" Jul 06 21:42:52.326755 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.326730 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6jnl\" (UniqueName: \"kubernetes.io/projected/639574ec-8467-4bb3-9203-6168444f8331-kube-api-access-m6jnl\") pod \"error-404-isvc-974ed-predictor-677df94fc9-c69t8\" (UID: \"639574ec-8467-4bb3-9203-6168444f8331\") " pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" Jul 06 21:42:52.370170 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.370141 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" Jul 06 21:42:52.440386 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.440355 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" Jul 06 21:42:52.507891 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.507808 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd"] Jul 06 21:42:52.510416 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:42:52.510387 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0d4bf0_2bec_413c_98d1_e3a5f264ab26.slice/crio-fd73a8ff07434ee343b1d5608aa0973af041fbade5ec0b11f1afebaacc774fff WatchSource:0}: Error finding container fd73a8ff07434ee343b1d5608aa0973af041fbade5ec0b11f1afebaacc774fff: Status 404 returned error can't find the container with id fd73a8ff07434ee343b1d5608aa0973af041fbade5ec0b11f1afebaacc774fff Jul 06 21:42:52.564905 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.564873 2581 generic.go:356] "Generic (PLEG): container finished" podID="919fd4d5-49d6-4638-b938-4285e9cce70d" containerID="a3038ba06b576c81bcb8ec3d07c41490561cf0ae2bd12adce659f1481e04fd4e" exitCode=2 Jul 06 21:42:52.565052 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.564910 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" event={"ID":"919fd4d5-49d6-4638-b938-4285e9cce70d","Type":"ContainerDied","Data":"a3038ba06b576c81bcb8ec3d07c41490561cf0ae2bd12adce659f1481e04fd4e"} Jul 06 21:42:52.566464 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.566435 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" event={"ID":"1f0d4bf0-2bec-413c-98d1-e3a5f264ab26","Type":"ContainerStarted","Data":"fd73a8ff07434ee343b1d5608aa0973af041fbade5ec0b11f1afebaacc774fff"} Jul 06 21:42:52.568197 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.568145 2581 generic.go:356] "Generic (PLEG): container finished" podID="9dc8783d-9353-4694-982e-febf4d48ee85" containerID="9720f2a5dd1ab6342980ae39a333ed875f3c2032199c73d2d06a8b7edd4fc0e3" exitCode=2 Jul 06 21:42:52.568197 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.568175 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" event={"ID":"9dc8783d-9353-4694-982e-febf4d48ee85","Type":"ContainerDied","Data":"9720f2a5dd1ab6342980ae39a333ed875f3c2032199c73d2d06a8b7edd4fc0e3"} Jul 06 21:42:52.586478 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:52.586454 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8"] Jul 06 21:42:52.589680 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:42:52.589657 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639574ec_8467_4bb3_9203_6168444f8331.slice/crio-fbbb754984da16696f843aeb723cc2521a190ab12a2c604c59efbef6141f6772 WatchSource:0}: Error finding container fbbb754984da16696f843aeb723cc2521a190ab12a2c604c59efbef6141f6772: Status 404 returned error can't find the container with id fbbb754984da16696f843aeb723cc2521a190ab12a2c604c59efbef6141f6772 Jul 06 21:42:53.574175 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:53.574132 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" event={"ID":"1f0d4bf0-2bec-413c-98d1-e3a5f264ab26","Type":"ContainerStarted","Data":"37308f37bc0765f228200fd64fa35ddb1ca252dc917ccd6bae4af0a9e67af3ea"} Jul 06 21:42:53.574654 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:53.574207 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" event={"ID":"1f0d4bf0-2bec-413c-98d1-e3a5f264ab26","Type":"ContainerStarted","Data":"e82962fbbe527837b1f45179531d3ad3008a9765ab7c92fd405b6b1e779690e5"} Jul 06 21:42:53.574654 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:53.574241 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" Jul 06 21:42:53.574654 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:53.574270 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" Jul 06 21:42:53.575683 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:53.575652 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" podUID="1f0d4bf0-2bec-413c-98d1-e3a5f264ab26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Jul 06 21:42:53.575922 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:53.575896 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" event={"ID":"639574ec-8467-4bb3-9203-6168444f8331","Type":"ContainerStarted","Data":"3ee7c6bc977a21cae874272ac6098360909dbe7dd44f80acf94fc2db1fb3fdf7"} Jul 06 21:42:53.575979 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:53.575948 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" event={"ID":"639574ec-8467-4bb3-9203-6168444f8331","Type":"ContainerStarted","Data":"864c1e72455b16155eba370a7746c2c72f59f25c410be4a102a8178b39486273"} Jul 06 21:42:53.575979 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:53.575974 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" event={"ID":"639574ec-8467-4bb3-9203-6168444f8331","Type":"ContainerStarted","Data":"fbbb754984da16696f843aeb723cc2521a190ab12a2c604c59efbef6141f6772"} Jul 06 21:42:53.576051 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:53.576000 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" Jul 06 21:42:53.576051 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:53.576022 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" Jul 06 21:42:53.576908 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:53.576884 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" podUID="639574ec-8467-4bb3-9203-6168444f8331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Jul 06 21:42:53.592192 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:53.592154 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" podStartSLOduration=1.592143123 podStartE2EDuration="1.592143123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:42:53.591082131 +0000 UTC m=+1798.353995147" watchObservedRunningTime="2026-07-06 21:42:53.592143123 +0000 UTC m=+1798.355056127" Jul 06 21:42:53.607921 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:53.607878 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" podStartSLOduration=1.6078673989999999 podStartE2EDuration="1.607867399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:42:53.607159084 +0000 UTC m=+1798.370072089" watchObservedRunningTime="2026-07-06 21:42:53.607867399 +0000 UTC m=+1798.370780404" Jul 06 21:42:54.140966 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:54.140922 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" podUID="9dc8783d-9353-4694-982e-febf4d48ee85" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused" Jul 06 21:42:54.149182 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:54.149141 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" podUID="9dc8783d-9353-4694-982e-febf4d48ee85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Jul 06 21:42:54.582966 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:54.582922 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" podUID="1f0d4bf0-2bec-413c-98d1-e3a5f264ab26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Jul 06 21:42:54.582966 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:54.582965 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" podUID="639574ec-8467-4bb3-9203-6168444f8331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Jul 06 21:42:55.145681 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.145633 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" podUID="919fd4d5-49d6-4638-b938-4285e9cce70d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused" Jul 06 21:42:55.151001 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.150957 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" podUID="919fd4d5-49d6-4638-b938-4285e9cce70d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Jul 06 21:42:55.309353 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.309316 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" Jul 06 21:42:55.343297 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.343263 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9dc8783d-9353-4694-982e-febf4d48ee85-proxy-tls\") pod \"9dc8783d-9353-4694-982e-febf4d48ee85\" (UID: \"9dc8783d-9353-4694-982e-febf4d48ee85\") " Jul 06 21:42:55.343483 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.343355 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8djk\" (UniqueName: \"kubernetes.io/projected/9dc8783d-9353-4694-982e-febf4d48ee85-kube-api-access-q8djk\") pod \"9dc8783d-9353-4694-982e-febf4d48ee85\" (UID: \"9dc8783d-9353-4694-982e-febf4d48ee85\") " Jul 06 21:42:55.343483 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.343426 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9dc8783d-9353-4694-982e-febf4d48ee85-kube-rbac-proxy-sar-config\") pod \"9dc8783d-9353-4694-982e-febf4d48ee85\" (UID: \"9dc8783d-9353-4694-982e-febf4d48ee85\") " Jul 06 21:42:55.343793 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.343768 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc8783d-9353-4694-982e-febf4d48ee85-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9dc8783d-9353-4694-982e-febf4d48ee85" (UID: "9dc8783d-9353-4694-982e-febf4d48ee85"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:42:55.345498 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.345465 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc8783d-9353-4694-982e-febf4d48ee85-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9dc8783d-9353-4694-982e-febf4d48ee85" (UID: "9dc8783d-9353-4694-982e-febf4d48ee85"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:42:55.345498 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.345474 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc8783d-9353-4694-982e-febf4d48ee85-kube-api-access-q8djk" (OuterVolumeSpecName: "kube-api-access-q8djk") pod "9dc8783d-9353-4694-982e-febf4d48ee85" (UID: "9dc8783d-9353-4694-982e-febf4d48ee85"). InnerVolumeSpecName "kube-api-access-q8djk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:42:55.445000 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.444969 2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9dc8783d-9353-4694-982e-febf4d48ee85-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:42:55.445119 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.445004 2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9dc8783d-9353-4694-982e-febf4d48ee85-proxy-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:42:55.445119 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.445021 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8djk\" (UniqueName: \"kubernetes.io/projected/9dc8783d-9353-4694-982e-febf4d48ee85-kube-api-access-q8djk\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:42:55.507492 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.507467 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" Jul 06 21:42:55.545943 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.545902 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxkzb\" (UniqueName: \"kubernetes.io/projected/919fd4d5-49d6-4638-b938-4285e9cce70d-kube-api-access-sxkzb\") pod \"919fd4d5-49d6-4638-b938-4285e9cce70d\" (UID: \"919fd4d5-49d6-4638-b938-4285e9cce70d\") " Jul 06 21:42:55.546105 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.545962 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/919fd4d5-49d6-4638-b938-4285e9cce70d-proxy-tls\") pod \"919fd4d5-49d6-4638-b938-4285e9cce70d\" (UID: \"919fd4d5-49d6-4638-b938-4285e9cce70d\") " Jul 06 21:42:55.546150 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.546107 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/919fd4d5-49d6-4638-b938-4285e9cce70d-kube-rbac-proxy-sar-config\") pod \"919fd4d5-49d6-4638-b938-4285e9cce70d\" (UID: \"919fd4d5-49d6-4638-b938-4285e9cce70d\") " Jul 06 21:42:55.546498 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.546468 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/919fd4d5-49d6-4638-b938-4285e9cce70d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "919fd4d5-49d6-4638-b938-4285e9cce70d" (UID: "919fd4d5-49d6-4638-b938-4285e9cce70d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:42:55.548095 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.548065 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919fd4d5-49d6-4638-b938-4285e9cce70d-kube-api-access-sxkzb" (OuterVolumeSpecName: "kube-api-access-sxkzb") pod "919fd4d5-49d6-4638-b938-4285e9cce70d" (UID: "919fd4d5-49d6-4638-b938-4285e9cce70d"). InnerVolumeSpecName "kube-api-access-sxkzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:42:55.548216 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.548196 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/919fd4d5-49d6-4638-b938-4285e9cce70d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "919fd4d5-49d6-4638-b938-4285e9cce70d" (UID: "919fd4d5-49d6-4638-b938-4285e9cce70d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:42:55.588482 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.588447 2581 generic.go:356] "Generic (PLEG): container finished" podID="919fd4d5-49d6-4638-b938-4285e9cce70d" containerID="c90c86031c7aefac581108b58bdffd1900ff461e47acee7d74b4fdbfa6e3b696" exitCode=0 Jul 06 21:42:55.588940 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.588500 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" Jul 06 21:42:55.588940 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.588537 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" event={"ID":"919fd4d5-49d6-4638-b938-4285e9cce70d","Type":"ContainerDied","Data":"c90c86031c7aefac581108b58bdffd1900ff461e47acee7d74b4fdbfa6e3b696"} Jul 06 21:42:55.588940 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.588597 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx" event={"ID":"919fd4d5-49d6-4638-b938-4285e9cce70d","Type":"ContainerDied","Data":"e6f3440699f8dda1101bd97c0c415f035ea39c13a0c1abcf93bab7b2a60b587c"} Jul 06 21:42:55.588940 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.588632 2581 scope.go:117] "RemoveContainer" containerID="a3038ba06b576c81bcb8ec3d07c41490561cf0ae2bd12adce659f1481e04fd4e" Jul 06 21:42:55.590330 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.590304 2581 generic.go:356] "Generic (PLEG): container finished" podID="9dc8783d-9353-4694-982e-febf4d48ee85" containerID="dc5d88742458cb07440b471f2241d7619fac6180864125e9938dc6da5b805497" exitCode=0 Jul 06 21:42:55.590461 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.590380 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" event={"ID":"9dc8783d-9353-4694-982e-febf4d48ee85","Type":"ContainerDied","Data":"dc5d88742458cb07440b471f2241d7619fac6180864125e9938dc6da5b805497"} Jul 06 21:42:55.590461 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.590454 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" event={"ID":"9dc8783d-9353-4694-982e-febf4d48ee85","Type":"ContainerDied","Data":"c7bcde0cbcd946532b338f186cfcb3259f2152bced2f96f27be69d39bd812acf"} Jul 06 21:42:55.590557 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.590391 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp" Jul 06 21:42:55.598154 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.598136 2581 scope.go:117] "RemoveContainer" containerID="c90c86031c7aefac581108b58bdffd1900ff461e47acee7d74b4fdbfa6e3b696" Jul 06 21:42:55.606249 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.606232 2581 scope.go:117] "RemoveContainer" containerID="a3038ba06b576c81bcb8ec3d07c41490561cf0ae2bd12adce659f1481e04fd4e" Jul 06 21:42:55.606506 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:42:55.606488 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3038ba06b576c81bcb8ec3d07c41490561cf0ae2bd12adce659f1481e04fd4e\": container with ID starting with a3038ba06b576c81bcb8ec3d07c41490561cf0ae2bd12adce659f1481e04fd4e not found: ID does not exist" containerID="a3038ba06b576c81bcb8ec3d07c41490561cf0ae2bd12adce659f1481e04fd4e" Jul 06 21:42:55.606590 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.606518 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3038ba06b576c81bcb8ec3d07c41490561cf0ae2bd12adce659f1481e04fd4e"} err="failed to get container status \"a3038ba06b576c81bcb8ec3d07c41490561cf0ae2bd12adce659f1481e04fd4e\": rpc error: code = NotFound desc = could not find container \"a3038ba06b576c81bcb8ec3d07c41490561cf0ae2bd12adce659f1481e04fd4e\": container with ID starting with a3038ba06b576c81bcb8ec3d07c41490561cf0ae2bd12adce659f1481e04fd4e not found: ID does not exist" Jul 06 21:42:55.606590 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.606540 2581 scope.go:117] "RemoveContainer" containerID="c90c86031c7aefac581108b58bdffd1900ff461e47acee7d74b4fdbfa6e3b696" Jul 06 21:42:55.606808 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:42:55.606791 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c90c86031c7aefac581108b58bdffd1900ff461e47acee7d74b4fdbfa6e3b696\": container with ID starting with c90c86031c7aefac581108b58bdffd1900ff461e47acee7d74b4fdbfa6e3b696 not found: ID does not exist" containerID="c90c86031c7aefac581108b58bdffd1900ff461e47acee7d74b4fdbfa6e3b696" Jul 06 21:42:55.606850 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.606814 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c90c86031c7aefac581108b58bdffd1900ff461e47acee7d74b4fdbfa6e3b696"} err="failed to get container status \"c90c86031c7aefac581108b58bdffd1900ff461e47acee7d74b4fdbfa6e3b696\": rpc error: code = NotFound desc = could not find container \"c90c86031c7aefac581108b58bdffd1900ff461e47acee7d74b4fdbfa6e3b696\": container with ID starting with c90c86031c7aefac581108b58bdffd1900ff461e47acee7d74b4fdbfa6e3b696 not found: ID does not exist" Jul 06 21:42:55.606850 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.606831 2581 scope.go:117] "RemoveContainer" containerID="9720f2a5dd1ab6342980ae39a333ed875f3c2032199c73d2d06a8b7edd4fc0e3" Jul 06 21:42:55.613819 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.613795 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx"] Jul 06 21:42:55.614743 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.614704 2581 scope.go:117] "RemoveContainer" containerID="dc5d88742458cb07440b471f2241d7619fac6180864125e9938dc6da5b805497" Jul 06 21:42:55.617153 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.617133 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9ded0-predictor-769fb7bc7-d9fvx"] Jul 06 21:42:55.622109 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.622092 2581 scope.go:117] "RemoveContainer" containerID="9720f2a5dd1ab6342980ae39a333ed875f3c2032199c73d2d06a8b7edd4fc0e3" Jul 06 21:42:55.622401 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:42:55.622381 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9720f2a5dd1ab6342980ae39a333ed875f3c2032199c73d2d06a8b7edd4fc0e3\": container with ID starting with 9720f2a5dd1ab6342980ae39a333ed875f3c2032199c73d2d06a8b7edd4fc0e3 not found: ID does not exist" containerID="9720f2a5dd1ab6342980ae39a333ed875f3c2032199c73d2d06a8b7edd4fc0e3" Jul 06 21:42:55.622481 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.622405 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9720f2a5dd1ab6342980ae39a333ed875f3c2032199c73d2d06a8b7edd4fc0e3"} err="failed to get container status \"9720f2a5dd1ab6342980ae39a333ed875f3c2032199c73d2d06a8b7edd4fc0e3\": rpc error: code = NotFound desc = could not find container \"9720f2a5dd1ab6342980ae39a333ed875f3c2032199c73d2d06a8b7edd4fc0e3\": container with ID starting with 9720f2a5dd1ab6342980ae39a333ed875f3c2032199c73d2d06a8b7edd4fc0e3 not found: ID does not exist" Jul 06 21:42:55.622481 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.622421 2581 scope.go:117] "RemoveContainer" containerID="dc5d88742458cb07440b471f2241d7619fac6180864125e9938dc6da5b805497" Jul 06 21:42:55.622689 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:42:55.622665 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc5d88742458cb07440b471f2241d7619fac6180864125e9938dc6da5b805497\": container with ID starting with dc5d88742458cb07440b471f2241d7619fac6180864125e9938dc6da5b805497 not found: ID does not exist" containerID="dc5d88742458cb07440b471f2241d7619fac6180864125e9938dc6da5b805497" Jul 06 21:42:55.622734 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.622696 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc5d88742458cb07440b471f2241d7619fac6180864125e9938dc6da5b805497"} err="failed to get container status \"dc5d88742458cb07440b471f2241d7619fac6180864125e9938dc6da5b805497\": rpc error: code = NotFound desc = could not find container \"dc5d88742458cb07440b471f2241d7619fac6180864125e9938dc6da5b805497\": container with ID starting with dc5d88742458cb07440b471f2241d7619fac6180864125e9938dc6da5b805497 not found: ID does not exist" Jul 06 21:42:55.627212 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.627191 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp"] Jul 06 21:42:55.633153 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.633133 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9ded0-predictor-6fc78767d4-9nnsp"] Jul 06 21:42:55.646826 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.646801 2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/919fd4d5-49d6-4638-b938-4285e9cce70d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:42:55.646899 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.646832 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxkzb\" (UniqueName: \"kubernetes.io/projected/919fd4d5-49d6-4638-b938-4285e9cce70d-kube-api-access-sxkzb\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:42:55.646899 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.646848 2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/919fd4d5-49d6-4638-b938-4285e9cce70d-proxy-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:42:55.787459 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.787369 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="919fd4d5-49d6-4638-b938-4285e9cce70d" path="/var/lib/kubelet/pods/919fd4d5-49d6-4638-b938-4285e9cce70d/volumes" Jul 06 21:42:55.787920 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:55.787900 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dc8783d-9353-4694-982e-febf4d48ee85" path="/var/lib/kubelet/pods/9dc8783d-9353-4694-982e-febf4d48ee85/volumes" Jul 06 21:42:59.588294 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:59.588257 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" Jul 06 21:42:59.589179 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:59.589150 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" podUID="1f0d4bf0-2bec-413c-98d1-e3a5f264ab26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Jul 06 21:42:59.589524 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:59.589502 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" Jul 06 21:42:59.589922 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:42:59.589902 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" podUID="639574ec-8467-4bb3-9203-6168444f8331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Jul 06 21:43:09.589844 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:09.589796 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" podUID="1f0d4bf0-2bec-413c-98d1-e3a5f264ab26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Jul 06 21:43:09.590236 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:09.589917 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" podUID="639574ec-8467-4bb3-9203-6168444f8331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Jul 06 21:43:19.589472 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:19.589430 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" podUID="1f0d4bf0-2bec-413c-98d1-e3a5f264ab26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Jul 06 21:43:19.589917 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:19.589892 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" podUID="639574ec-8467-4bb3-9203-6168444f8331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Jul 06 21:43:29.589490 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:29.589450 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" podUID="1f0d4bf0-2bec-413c-98d1-e3a5f264ab26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Jul 06 21:43:29.589988 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:29.589860 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" podUID="639574ec-8467-4bb3-9203-6168444f8331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Jul 06 21:43:39.590538 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:39.590505 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" Jul 06 21:43:39.591000 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:39.590661 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" Jul 06 21:43:51.904089 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.904053 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb"] Jul 06 21:43:51.904530 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.904383 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" podUID="faad73e2-ca54-4e10-8af0-003bbc901de6" containerName="kserve-container" containerID="cri-o://4dcc035d10208605a3ffa3ac1af744e8e4518352ee0f943b5bdad1a6d85ee4e5" gracePeriod=30 Jul 06 21:43:51.904530 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.904446 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" podUID="faad73e2-ca54-4e10-8af0-003bbc901de6" containerName="kube-rbac-proxy" containerID="cri-o://b836709a3a02c823e5354a0c729d2b2e53c6a79015a1c596f901719aa60b7255" gracePeriod=30 Jul 06 21:43:51.930111 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.930079 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx"] Jul 06 21:43:51.930474 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.930461 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="919fd4d5-49d6-4638-b938-4285e9cce70d" containerName="kserve-container" Jul 06 21:43:51.930523 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.930476 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="919fd4d5-49d6-4638-b938-4285e9cce70d" containerName="kserve-container" Jul 06 21:43:51.930523 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.930484 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="919fd4d5-49d6-4638-b938-4285e9cce70d" containerName="kube-rbac-proxy" Jul 06 21:43:51.930523 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.930489 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="919fd4d5-49d6-4638-b938-4285e9cce70d" containerName="kube-rbac-proxy" Jul 06 21:43:51.930523 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.930499 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dc8783d-9353-4694-982e-febf4d48ee85" containerName="kube-rbac-proxy" Jul 06 21:43:51.930523 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.930505 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dc8783d-9353-4694-982e-febf4d48ee85" containerName="kube-rbac-proxy" Jul 06 21:43:51.930523 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.930524 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dc8783d-9353-4694-982e-febf4d48ee85" containerName="kserve-container" Jul 06 21:43:51.930709 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.930529 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dc8783d-9353-4694-982e-febf4d48ee85" containerName="kserve-container" Jul 06 21:43:51.930709 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.930582 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dc8783d-9353-4694-982e-febf4d48ee85" containerName="kube-rbac-proxy" Jul 06 21:43:51.930709 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.930589 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dc8783d-9353-4694-982e-febf4d48ee85" containerName="kserve-container" Jul 06 21:43:51.930709 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.930596 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="919fd4d5-49d6-4638-b938-4285e9cce70d" containerName="kube-rbac-proxy" Jul 06 21:43:51.930709 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.930604 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="919fd4d5-49d6-4638-b938-4285e9cce70d" containerName="kserve-container" Jul 06 21:43:51.933731 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.933713 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" Jul 06 21:43:51.936087 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.936065 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-da1f6-kube-rbac-proxy-sar-config\"" Jul 06 21:43:51.936243 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.936224 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-da1f6-predictor-serving-cert\"" Jul 06 21:43:51.942515 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.942490 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx"] Jul 06 21:43:51.993031 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.992993 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh"] Jul 06 21:43:51.993328 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.993275 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" podUID="7830a974-68bd-462d-addc-faad0a1735d5" containerName="kserve-container" containerID="cri-o://07c882e08eab3e813db106965f38729872fcc4ffa3e8788d8aa726a85232f44f" gracePeriod=30 Jul 06 21:43:51.993433 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:51.993314 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" podUID="7830a974-68bd-462d-addc-faad0a1735d5" containerName="kube-rbac-proxy" containerID="cri-o://188a4de7e53802852ea047fbdc2872e05636e9f36f34fcb6a54232539f7240c8" gracePeriod=30 Jul 06 21:43:52.029418 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.029327 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj"] Jul 06 21:43:52.032784 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.032761 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" Jul 06 21:43:52.035112 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.035080 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-da1f6-predictor-serving-cert\"" Jul 06 21:43:52.035240 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.035179 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-da1f6-kube-rbac-proxy-sar-config\"" Jul 06 21:43:52.038370 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.038333 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5913d04b-be39-4226-8825-68074a5e74ea-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx\" (UID: \"5913d04b-be39-4226-8825-68074a5e74ea\") " pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" Jul 06 21:43:52.038490 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.038444 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5913d04b-be39-4226-8825-68074a5e74ea-proxy-tls\") pod \"success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx\" (UID: \"5913d04b-be39-4226-8825-68074a5e74ea\") " pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" Jul 06 21:43:52.038553 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.038516 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8qqt\" (UniqueName: \"kubernetes.io/projected/5913d04b-be39-4226-8825-68074a5e74ea-kube-api-access-b8qqt\") pod \"success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx\" (UID: \"5913d04b-be39-4226-8825-68074a5e74ea\") " pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" Jul 06 21:43:52.044213 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.044191 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj"] Jul 06 21:43:52.139369 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.139303 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f20475b7-1ad4-43b0-b506-e72e28b106f4-proxy-tls\") pod \"error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj\" (UID: \"f20475b7-1ad4-43b0-b506-e72e28b106f4\") " pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" Jul 06 21:43:52.139537 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.139363 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8qqt\" (UniqueName: \"kubernetes.io/projected/5913d04b-be39-4226-8825-68074a5e74ea-kube-api-access-b8qqt\") pod \"success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx\" (UID: \"5913d04b-be39-4226-8825-68074a5e74ea\") " pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" Jul 06 21:43:52.139537 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.139435 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqhjk\" (UniqueName: \"kubernetes.io/projected/f20475b7-1ad4-43b0-b506-e72e28b106f4-kube-api-access-pqhjk\") pod \"error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj\" (UID: \"f20475b7-1ad4-43b0-b506-e72e28b106f4\") " pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" Jul 06 21:43:52.139537 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.139462 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f20475b7-1ad4-43b0-b506-e72e28b106f4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj\" (UID: \"f20475b7-1ad4-43b0-b506-e72e28b106f4\") " pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" Jul 06 21:43:52.139537 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.139495 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5913d04b-be39-4226-8825-68074a5e74ea-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx\" (UID: \"5913d04b-be39-4226-8825-68074a5e74ea\") " pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" Jul 06 21:43:52.139720 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.139586 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5913d04b-be39-4226-8825-68074a5e74ea-proxy-tls\") pod \"success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx\" (UID: \"5913d04b-be39-4226-8825-68074a5e74ea\") " pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" Jul 06 21:43:52.140256 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.140233 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5913d04b-be39-4226-8825-68074a5e74ea-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx\" (UID: \"5913d04b-be39-4226-8825-68074a5e74ea\") " pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" Jul 06 21:43:52.142036 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.142006 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5913d04b-be39-4226-8825-68074a5e74ea-proxy-tls\") pod \"success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx\" (UID: \"5913d04b-be39-4226-8825-68074a5e74ea\") " pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" Jul 06 21:43:52.149785 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.149754 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8qqt\" (UniqueName: \"kubernetes.io/projected/5913d04b-be39-4226-8825-68074a5e74ea-kube-api-access-b8qqt\") pod \"success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx\" (UID: \"5913d04b-be39-4226-8825-68074a5e74ea\") " pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" Jul 06 21:43:52.240969 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.240870 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqhjk\" (UniqueName: \"kubernetes.io/projected/f20475b7-1ad4-43b0-b506-e72e28b106f4-kube-api-access-pqhjk\") pod \"error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj\" (UID: \"f20475b7-1ad4-43b0-b506-e72e28b106f4\") " pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" Jul 06 21:43:52.240969 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.240913 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f20475b7-1ad4-43b0-b506-e72e28b106f4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj\" (UID: \"f20475b7-1ad4-43b0-b506-e72e28b106f4\") " pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" Jul 06 21:43:52.241221 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.241004 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f20475b7-1ad4-43b0-b506-e72e28b106f4-proxy-tls\") pod \"error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj\" (UID: \"f20475b7-1ad4-43b0-b506-e72e28b106f4\") " pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" Jul 06 21:43:52.241626 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.241603 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f20475b7-1ad4-43b0-b506-e72e28b106f4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj\" (UID: \"f20475b7-1ad4-43b0-b506-e72e28b106f4\") " pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" Jul 06 21:43:52.243472 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.243447 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f20475b7-1ad4-43b0-b506-e72e28b106f4-proxy-tls\") pod \"error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj\" (UID: \"f20475b7-1ad4-43b0-b506-e72e28b106f4\") " pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" Jul 06 21:43:52.245310 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.245285 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" Jul 06 21:43:52.248682 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.248658 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqhjk\" (UniqueName: \"kubernetes.io/projected/f20475b7-1ad4-43b0-b506-e72e28b106f4-kube-api-access-pqhjk\") pod \"error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj\" (UID: \"f20475b7-1ad4-43b0-b506-e72e28b106f4\") " pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" Jul 06 21:43:52.346564 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.346527 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" Jul 06 21:43:52.383607 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.383571 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx"] Jul 06 21:43:52.385085 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:43:52.385058 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5913d04b_be39_4226_8825_68074a5e74ea.slice/crio-1dffc05077687776c668b23991ca8c8a3feb7cf93eeaa3127939b61b31e99d33 WatchSource:0}: Error finding container 1dffc05077687776c668b23991ca8c8a3feb7cf93eeaa3127939b61b31e99d33: Status 404 returned error can't find the container with id 1dffc05077687776c668b23991ca8c8a3feb7cf93eeaa3127939b61b31e99d33 Jul 06 21:43:52.490688 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.490639 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj"] Jul 06 21:43:52.493274 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:43:52.493218 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20475b7_1ad4_43b0_b506_e72e28b106f4.slice/crio-6380d58be1ceab0bb7fa3cc62bf69a9fc7aac59b852411cede41f27e1954094e WatchSource:0}: Error finding container 6380d58be1ceab0bb7fa3cc62bf69a9fc7aac59b852411cede41f27e1954094e: Status 404 returned error can't find the container with id 6380d58be1ceab0bb7fa3cc62bf69a9fc7aac59b852411cede41f27e1954094e Jul 06 21:43:52.826111 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.826018 2581 generic.go:356] "Generic (PLEG): container finished" podID="faad73e2-ca54-4e10-8af0-003bbc901de6" containerID="b836709a3a02c823e5354a0c729d2b2e53c6a79015a1c596f901719aa60b7255" exitCode=2 Jul 06 21:43:52.826111 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.826087 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" event={"ID":"faad73e2-ca54-4e10-8af0-003bbc901de6","Type":"ContainerDied","Data":"b836709a3a02c823e5354a0c729d2b2e53c6a79015a1c596f901719aa60b7255"} Jul 06 21:43:52.827791 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.827769 2581 generic.go:356] "Generic (PLEG): container finished" podID="7830a974-68bd-462d-addc-faad0a1735d5" containerID="188a4de7e53802852ea047fbdc2872e05636e9f36f34fcb6a54232539f7240c8" exitCode=2 Jul 06 21:43:52.827927 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.827842 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" event={"ID":"7830a974-68bd-462d-addc-faad0a1735d5","Type":"ContainerDied","Data":"188a4de7e53802852ea047fbdc2872e05636e9f36f34fcb6a54232539f7240c8"} Jul 06 21:43:52.829503 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.829480 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" event={"ID":"f20475b7-1ad4-43b0-b506-e72e28b106f4","Type":"ContainerStarted","Data":"c89ce198514bd851b421357f74efa0677d1b82abed6c95fc7473475e85113278"} Jul 06 21:43:52.829645 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.829525 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" event={"ID":"f20475b7-1ad4-43b0-b506-e72e28b106f4","Type":"ContainerStarted","Data":"00047d17bbad2f47c8cfc2ab2a8bf0dab9c2eb5b9ced0b4d0fa30f090baff9d3"} Jul 06 21:43:52.829645 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.829552 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" event={"ID":"f20475b7-1ad4-43b0-b506-e72e28b106f4","Type":"ContainerStarted","Data":"6380d58be1ceab0bb7fa3cc62bf69a9fc7aac59b852411cede41f27e1954094e"} Jul 06 21:43:52.829645 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.829607 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" Jul 06 21:43:52.831134 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.831115 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" event={"ID":"5913d04b-be39-4226-8825-68074a5e74ea","Type":"ContainerStarted","Data":"5bb1ac559587d481cb73ab3f0d5a69a271e2a8f279501dd332629792bb598bd7"} Jul 06 21:43:52.831241 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.831156 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" event={"ID":"5913d04b-be39-4226-8825-68074a5e74ea","Type":"ContainerStarted","Data":"865553cc212221dfc0a2512e3fa2e0fc000c515b403bcec127d87c052870b5a8"} Jul 06 21:43:52.831241 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.831177 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" event={"ID":"5913d04b-be39-4226-8825-68074a5e74ea","Type":"ContainerStarted","Data":"1dffc05077687776c668b23991ca8c8a3feb7cf93eeaa3127939b61b31e99d33"} Jul 06 21:43:52.831241 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.831203 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" Jul 06 21:43:52.831413 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.831262 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" Jul 06 21:43:52.832854 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.832831 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" podUID="5913d04b-be39-4226-8825-68074a5e74ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Jul 06 21:43:52.845839 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.845793 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" podStartSLOduration=0.845777169 podStartE2EDuration="845.777169ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:43:52.84500896 +0000 UTC m=+1857.607921975" watchObservedRunningTime="2026-07-06 21:43:52.845777169 +0000 UTC m=+1857.608690176" Jul 06 21:43:52.863211 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:52.863159 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" podStartSLOduration=1.863141854 podStartE2EDuration="1.863141854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:43:52.860917961 +0000 UTC m=+1857.623830964" watchObservedRunningTime="2026-07-06 21:43:52.863141854 +0000 UTC m=+1857.626054953" Jul 06 21:43:53.835428 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:53.835389 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" podUID="5913d04b-be39-4226-8825-68074a5e74ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Jul 06 21:43:53.835931 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:53.835520 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" Jul 06 21:43:53.836840 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:53.836813 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" podUID="f20475b7-1ad4-43b0-b506-e72e28b106f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Jul 06 21:43:54.478759 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:54.478712 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" podUID="7830a974-68bd-462d-addc-faad0a1735d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused" Jul 06 21:43:54.483048 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:54.483017 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" podUID="7830a974-68bd-462d-addc-faad0a1735d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Jul 06 21:43:54.840367 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:54.840247 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" podUID="f20475b7-1ad4-43b0-b506-e72e28b106f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Jul 06 21:43:55.356667 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.356631 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" Jul 06 21:43:55.470171 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.470077 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7830a974-68bd-462d-addc-faad0a1735d5-proxy-tls\") pod \"7830a974-68bd-462d-addc-faad0a1735d5\" (UID: \"7830a974-68bd-462d-addc-faad0a1735d5\") " Jul 06 21:43:55.470319 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.470206 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7830a974-68bd-462d-addc-faad0a1735d5-kube-rbac-proxy-sar-config\") pod \"7830a974-68bd-462d-addc-faad0a1735d5\" (UID: \"7830a974-68bd-462d-addc-faad0a1735d5\") " Jul 06 21:43:55.470319 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.470239 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckf8j\" (UniqueName: \"kubernetes.io/projected/7830a974-68bd-462d-addc-faad0a1735d5-kube-api-access-ckf8j\") pod \"7830a974-68bd-462d-addc-faad0a1735d5\" (UID: \"7830a974-68bd-462d-addc-faad0a1735d5\") " Jul 06 21:43:55.470608 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.470581 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7830a974-68bd-462d-addc-faad0a1735d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7830a974-68bd-462d-addc-faad0a1735d5" (UID: "7830a974-68bd-462d-addc-faad0a1735d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:43:55.472062 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.472039 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7830a974-68bd-462d-addc-faad0a1735d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7830a974-68bd-462d-addc-faad0a1735d5" (UID: "7830a974-68bd-462d-addc-faad0a1735d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:43:55.472214 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.472186 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7830a974-68bd-462d-addc-faad0a1735d5-kube-api-access-ckf8j" (OuterVolumeSpecName: "kube-api-access-ckf8j") pod "7830a974-68bd-462d-addc-faad0a1735d5" (UID: "7830a974-68bd-462d-addc-faad0a1735d5"). InnerVolumeSpecName "kube-api-access-ckf8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:43:55.484620 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.484589 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" podUID="faad73e2-ca54-4e10-8af0-003bbc901de6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused" Jul 06 21:43:55.489945 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.489919 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" podUID="faad73e2-ca54-4e10-8af0-003bbc901de6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 06 21:43:55.572012 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.571977 2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7830a974-68bd-462d-addc-faad0a1735d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:43:55.572139 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.572016 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckf8j\" (UniqueName: \"kubernetes.io/projected/7830a974-68bd-462d-addc-faad0a1735d5-kube-api-access-ckf8j\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:43:55.572139 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.572034 2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7830a974-68bd-462d-addc-faad0a1735d5-proxy-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:43:55.740786 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.740759 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" Jul 06 21:43:55.846584 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.846548 2581 generic.go:356] "Generic (PLEG): container finished" podID="7830a974-68bd-462d-addc-faad0a1735d5" containerID="07c882e08eab3e813db106965f38729872fcc4ffa3e8788d8aa726a85232f44f" exitCode=0 Jul 06 21:43:55.847013 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.846600 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" event={"ID":"7830a974-68bd-462d-addc-faad0a1735d5","Type":"ContainerDied","Data":"07c882e08eab3e813db106965f38729872fcc4ffa3e8788d8aa726a85232f44f"} Jul 06 21:43:55.847013 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.846664 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" event={"ID":"7830a974-68bd-462d-addc-faad0a1735d5","Type":"ContainerDied","Data":"e8955f32c6047075283a37a59639435765eddc24fa5a91758375ce6e730576c3"} Jul 06 21:43:55.847013 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.846620 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh" Jul 06 21:43:55.847013 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.846697 2581 scope.go:117] "RemoveContainer" containerID="188a4de7e53802852ea047fbdc2872e05636e9f36f34fcb6a54232539f7240c8" Jul 06 21:43:55.848499 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.848474 2581 generic.go:356] "Generic (PLEG): container finished" podID="faad73e2-ca54-4e10-8af0-003bbc901de6" containerID="4dcc035d10208605a3ffa3ac1af744e8e4518352ee0f943b5bdad1a6d85ee4e5" exitCode=0 Jul 06 21:43:55.848632 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.848534 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" Jul 06 21:43:55.848632 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.848560 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" event={"ID":"faad73e2-ca54-4e10-8af0-003bbc901de6","Type":"ContainerDied","Data":"4dcc035d10208605a3ffa3ac1af744e8e4518352ee0f943b5bdad1a6d85ee4e5"} Jul 06 21:43:55.848632 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.848611 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb" event={"ID":"faad73e2-ca54-4e10-8af0-003bbc901de6","Type":"ContainerDied","Data":"723d80c005bdc364755f36404a05b4f837360eee08f38505a4eb9a9231dfc77e"} Jul 06 21:43:55.856873 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.856841 2581 scope.go:117] "RemoveContainer" containerID="07c882e08eab3e813db106965f38729872fcc4ffa3e8788d8aa726a85232f44f" Jul 06 21:43:55.864464 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.864447 2581 scope.go:117] "RemoveContainer" containerID="188a4de7e53802852ea047fbdc2872e05636e9f36f34fcb6a54232539f7240c8" Jul 06 21:43:55.864882 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:43:55.864817 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"188a4de7e53802852ea047fbdc2872e05636e9f36f34fcb6a54232539f7240c8\": container with ID starting with 188a4de7e53802852ea047fbdc2872e05636e9f36f34fcb6a54232539f7240c8 not found: ID does not exist" containerID="188a4de7e53802852ea047fbdc2872e05636e9f36f34fcb6a54232539f7240c8" Jul 06 21:43:55.864990 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.864875 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"188a4de7e53802852ea047fbdc2872e05636e9f36f34fcb6a54232539f7240c8"} err="failed to get container status \"188a4de7e53802852ea047fbdc2872e05636e9f36f34fcb6a54232539f7240c8\": rpc error: code = NotFound desc = could not find container \"188a4de7e53802852ea047fbdc2872e05636e9f36f34fcb6a54232539f7240c8\": container with ID starting with 188a4de7e53802852ea047fbdc2872e05636e9f36f34fcb6a54232539f7240c8 not found: ID does not exist" Jul 06 21:43:55.864990 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.864899 2581 scope.go:117] "RemoveContainer" containerID="07c882e08eab3e813db106965f38729872fcc4ffa3e8788d8aa726a85232f44f" Jul 06 21:43:55.865681 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:43:55.865655 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c882e08eab3e813db106965f38729872fcc4ffa3e8788d8aa726a85232f44f\": container with ID starting with 07c882e08eab3e813db106965f38729872fcc4ffa3e8788d8aa726a85232f44f not found: ID does not exist" containerID="07c882e08eab3e813db106965f38729872fcc4ffa3e8788d8aa726a85232f44f" Jul 06 21:43:55.865880 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.865687 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c882e08eab3e813db106965f38729872fcc4ffa3e8788d8aa726a85232f44f"} err="failed to get container status \"07c882e08eab3e813db106965f38729872fcc4ffa3e8788d8aa726a85232f44f\": rpc error: code = NotFound desc = could not find container \"07c882e08eab3e813db106965f38729872fcc4ffa3e8788d8aa726a85232f44f\": container with ID starting with 07c882e08eab3e813db106965f38729872fcc4ffa3e8788d8aa726a85232f44f not found: ID does not exist" Jul 06 21:43:55.865880 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.865719 2581 scope.go:117] "RemoveContainer" containerID="b836709a3a02c823e5354a0c729d2b2e53c6a79015a1c596f901719aa60b7255" Jul 06 21:43:55.867303 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.867284 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh"] Jul 06 21:43:55.872476 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.872455 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a54dc-predictor-54fd4997ff-cbnmh"] Jul 06 21:43:55.873500 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.873484 2581 scope.go:117] "RemoveContainer" containerID="4dcc035d10208605a3ffa3ac1af744e8e4518352ee0f943b5bdad1a6d85ee4e5" Jul 06 21:43:55.874448 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.874427 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/faad73e2-ca54-4e10-8af0-003bbc901de6-kube-rbac-proxy-sar-config\") pod \"faad73e2-ca54-4e10-8af0-003bbc901de6\" (UID: \"faad73e2-ca54-4e10-8af0-003bbc901de6\") " Jul 06 21:43:55.874524 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.874514 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vst4w\" (UniqueName: \"kubernetes.io/projected/faad73e2-ca54-4e10-8af0-003bbc901de6-kube-api-access-vst4w\") pod \"faad73e2-ca54-4e10-8af0-003bbc901de6\" (UID: \"faad73e2-ca54-4e10-8af0-003bbc901de6\") " Jul 06 21:43:55.874583 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.874553 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/faad73e2-ca54-4e10-8af0-003bbc901de6-proxy-tls\") pod \"faad73e2-ca54-4e10-8af0-003bbc901de6\" (UID: \"faad73e2-ca54-4e10-8af0-003bbc901de6\") " Jul 06 21:43:55.874856 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.874829 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faad73e2-ca54-4e10-8af0-003bbc901de6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "faad73e2-ca54-4e10-8af0-003bbc901de6" (UID: "faad73e2-ca54-4e10-8af0-003bbc901de6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:43:55.876668 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.876639 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faad73e2-ca54-4e10-8af0-003bbc901de6-kube-api-access-vst4w" (OuterVolumeSpecName: "kube-api-access-vst4w") pod "faad73e2-ca54-4e10-8af0-003bbc901de6" (UID: "faad73e2-ca54-4e10-8af0-003bbc901de6"). InnerVolumeSpecName "kube-api-access-vst4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:43:55.876796 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.876771 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faad73e2-ca54-4e10-8af0-003bbc901de6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "faad73e2-ca54-4e10-8af0-003bbc901de6" (UID: "faad73e2-ca54-4e10-8af0-003bbc901de6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:43:55.878415 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.878395 2581 scope.go:117] "RemoveContainer" containerID="4dcc035d10208605a3ffa3ac1af744e8e4518352ee0f943b5bdad1a6d85ee4e5" Jul 06 21:43:55.881137 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.881120 2581 scope.go:117] "RemoveContainer" containerID="b836709a3a02c823e5354a0c729d2b2e53c6a79015a1c596f901719aa60b7255" Jul 06 21:43:55.881256 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:43:55.881129 2581 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_kserve-container_success-200-isvc-a54dc-predictor-558db86446-4lqqb_kserve-ci-e2e-test_faad73e2-ca54-4e10-8af0-003bbc901de6_0 in pod sandbox 723d80c005bdc364755f36404a05b4f837360eee08f38505a4eb9a9231dfc77e from index: no such id: '4dcc035d10208605a3ffa3ac1af744e8e4518352ee0f943b5bdad1a6d85ee4e5'" containerID="4dcc035d10208605a3ffa3ac1af744e8e4518352ee0f943b5bdad1a6d85ee4e5" Jul 06 21:43:55.881256 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:43:55.881165 2581 kuberuntime_gc.go:151] "Failed to remove container" err="rpc error: code = Unknown desc = failed to delete container k8s_kserve-container_success-200-isvc-a54dc-predictor-558db86446-4lqqb_kserve-ci-e2e-test_faad73e2-ca54-4e10-8af0-003bbc901de6_0 in pod sandbox 723d80c005bdc364755f36404a05b4f837360eee08f38505a4eb9a9231dfc77e from index: no such id: '4dcc035d10208605a3ffa3ac1af744e8e4518352ee0f943b5bdad1a6d85ee4e5'" containerID="4dcc035d10208605a3ffa3ac1af744e8e4518352ee0f943b5bdad1a6d85ee4e5" Jul 06 21:43:55.881476 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:43:55.881455 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b836709a3a02c823e5354a0c729d2b2e53c6a79015a1c596f901719aa60b7255\": container with ID starting with b836709a3a02c823e5354a0c729d2b2e53c6a79015a1c596f901719aa60b7255 not found: ID does not exist" containerID="b836709a3a02c823e5354a0c729d2b2e53c6a79015a1c596f901719aa60b7255" Jul 06 21:43:55.881552 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.881481 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b836709a3a02c823e5354a0c729d2b2e53c6a79015a1c596f901719aa60b7255"} err="failed to get container status \"b836709a3a02c823e5354a0c729d2b2e53c6a79015a1c596f901719aa60b7255\": rpc error: code = NotFound desc = could not find container \"b836709a3a02c823e5354a0c729d2b2e53c6a79015a1c596f901719aa60b7255\": container with ID starting with b836709a3a02c823e5354a0c729d2b2e53c6a79015a1c596f901719aa60b7255 not found: ID does not exist" Jul 06 21:43:55.881552 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.881499 2581 scope.go:117] "RemoveContainer" containerID="4dcc035d10208605a3ffa3ac1af744e8e4518352ee0f943b5bdad1a6d85ee4e5" Jul 06 21:43:55.881742 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:43:55.881724 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dcc035d10208605a3ffa3ac1af744e8e4518352ee0f943b5bdad1a6d85ee4e5\": container with ID starting with 4dcc035d10208605a3ffa3ac1af744e8e4518352ee0f943b5bdad1a6d85ee4e5 not found: ID does not exist" containerID="4dcc035d10208605a3ffa3ac1af744e8e4518352ee0f943b5bdad1a6d85ee4e5" Jul 06 21:43:55.881810 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.881747 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dcc035d10208605a3ffa3ac1af744e8e4518352ee0f943b5bdad1a6d85ee4e5"} err="failed to get container status \"4dcc035d10208605a3ffa3ac1af744e8e4518352ee0f943b5bdad1a6d85ee4e5\": rpc error: code = NotFound desc = could not find container \"4dcc035d10208605a3ffa3ac1af744e8e4518352ee0f943b5bdad1a6d85ee4e5\": container with ID starting with 4dcc035d10208605a3ffa3ac1af744e8e4518352ee0f943b5bdad1a6d85ee4e5 not found: ID does not exist" Jul 06 21:43:55.975973 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.975936 2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/faad73e2-ca54-4e10-8af0-003bbc901de6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:43:55.975973 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.975965 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vst4w\" (UniqueName: \"kubernetes.io/projected/faad73e2-ca54-4e10-8af0-003bbc901de6-kube-api-access-vst4w\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:43:55.975973 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:55.975975 2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/faad73e2-ca54-4e10-8af0-003bbc901de6-proxy-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:43:56.180287 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:56.180255 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb"] Jul 06 21:43:56.183940 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:56.183915 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a54dc-predictor-558db86446-4lqqb"] Jul 06 21:43:57.786590 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:57.786557 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7830a974-68bd-462d-addc-faad0a1735d5" path="/var/lib/kubelet/pods/7830a974-68bd-462d-addc-faad0a1735d5/volumes" Jul 06 21:43:57.787006 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:57.786992 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faad73e2-ca54-4e10-8af0-003bbc901de6" path="/var/lib/kubelet/pods/faad73e2-ca54-4e10-8af0-003bbc901de6/volumes" Jul 06 21:43:58.839770 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:58.839740 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" Jul 06 21:43:58.840224 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:58.840180 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" podUID="5913d04b-be39-4226-8825-68074a5e74ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Jul 06 21:43:59.845105 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:59.845075 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" Jul 06 21:43:59.845614 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:43:59.845583 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" podUID="f20475b7-1ad4-43b0-b506-e72e28b106f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Jul 06 21:44:02.320787 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.320746 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd"] Jul 06 21:44:02.321311 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.320962 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" podUID="1f0d4bf0-2bec-413c-98d1-e3a5f264ab26" containerName="kserve-container" containerID="cri-o://e82962fbbe527837b1f45179531d3ad3008a9765ab7c92fd405b6b1e779690e5" gracePeriod=30 Jul 06 21:44:02.321311 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.321004 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" podUID="1f0d4bf0-2bec-413c-98d1-e3a5f264ab26" containerName="kube-rbac-proxy" containerID="cri-o://37308f37bc0765f228200fd64fa35ddb1ca252dc917ccd6bae4af0a9e67af3ea" gracePeriod=30 Jul 06 21:44:02.376420 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.376385 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7"] Jul 06 21:44:02.376757 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.376745 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faad73e2-ca54-4e10-8af0-003bbc901de6" containerName="kube-rbac-proxy" Jul 06 21:44:02.376807 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.376759 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="faad73e2-ca54-4e10-8af0-003bbc901de6" containerName="kube-rbac-proxy" Jul 06 21:44:02.376807 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.376767 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7830a974-68bd-462d-addc-faad0a1735d5" containerName="kserve-container" Jul 06 21:44:02.376807 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.376772 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7830a974-68bd-462d-addc-faad0a1735d5" containerName="kserve-container" Jul 06 21:44:02.376807 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.376778 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7830a974-68bd-462d-addc-faad0a1735d5" containerName="kube-rbac-proxy" Jul 06 21:44:02.376807 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.376784 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7830a974-68bd-462d-addc-faad0a1735d5" containerName="kube-rbac-proxy" Jul 06 21:44:02.376807 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.376799 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faad73e2-ca54-4e10-8af0-003bbc901de6" containerName="kserve-container" Jul 06 21:44:02.376807 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.376804 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="faad73e2-ca54-4e10-8af0-003bbc901de6" containerName="kserve-container" Jul 06 21:44:02.377044 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.376859 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7830a974-68bd-462d-addc-faad0a1735d5" containerName="kube-rbac-proxy" Jul 06 21:44:02.377044 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.376869 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="faad73e2-ca54-4e10-8af0-003bbc901de6" containerName="kserve-container" Jul 06 21:44:02.377044 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.376875 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="faad73e2-ca54-4e10-8af0-003bbc901de6" containerName="kube-rbac-proxy" Jul 06 21:44:02.377044 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.376882 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7830a974-68bd-462d-addc-faad0a1735d5" containerName="kserve-container" Jul 06 21:44:02.379992 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.379973 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" Jul 06 21:44:02.382548 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.382527 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-563cb-kube-rbac-proxy-sar-config\"" Jul 06 21:44:02.382670 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.382576 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-563cb-predictor-serving-cert\"" Jul 06 21:44:02.391057 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.391031 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7"] Jul 06 21:44:02.423141 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.423110 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8"] Jul 06 21:44:02.423361 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.423318 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" podUID="639574ec-8467-4bb3-9203-6168444f8331" containerName="kserve-container" containerID="cri-o://864c1e72455b16155eba370a7746c2c72f59f25c410be4a102a8178b39486273" gracePeriod=30 Jul 06 21:44:02.423541 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.423376 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" podUID="639574ec-8467-4bb3-9203-6168444f8331" containerName="kube-rbac-proxy" containerID="cri-o://3ee7c6bc977a21cae874272ac6098360909dbe7dd44f80acf94fc2db1fb3fdf7" gracePeriod=30 Jul 06 21:44:02.475965 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.475928 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5"] Jul 06 21:44:02.479409 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.479390 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" Jul 06 21:44:02.481834 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.481814 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-563cb-predictor-serving-cert\"" Jul 06 21:44:02.481929 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.481821 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-563cb-kube-rbac-proxy-sar-config\"" Jul 06 21:44:02.489673 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.489651 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5"] Jul 06 21:44:02.529787 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.529756 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e6066cd-42c1-4c15-b699-a0a438779f53-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-563cb-predictor-59bfb879bb-x7zl7\" (UID: \"4e6066cd-42c1-4c15-b699-a0a438779f53\") " pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" Jul 06 21:44:02.529945 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.529803 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnrd6\" (UniqueName: \"kubernetes.io/projected/4e6066cd-42c1-4c15-b699-a0a438779f53-kube-api-access-bnrd6\") pod \"success-200-isvc-563cb-predictor-59bfb879bb-x7zl7\" (UID: \"4e6066cd-42c1-4c15-b699-a0a438779f53\") " pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" Jul 06 21:44:02.529945 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.529931 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e6066cd-42c1-4c15-b699-a0a438779f53-proxy-tls\") pod \"success-200-isvc-563cb-predictor-59bfb879bb-x7zl7\" (UID: \"4e6066cd-42c1-4c15-b699-a0a438779f53\") " pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" Jul 06 21:44:02.631172 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.631138 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76c95e95-e959-46eb-b3b6-aa7f979d615f-proxy-tls\") pod \"error-404-isvc-563cb-predictor-85497489cf-k8hg5\" (UID: \"76c95e95-e959-46eb-b3b6-aa7f979d615f\") " pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" Jul 06 21:44:02.631330 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.631189 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e6066cd-42c1-4c15-b699-a0a438779f53-proxy-tls\") pod \"success-200-isvc-563cb-predictor-59bfb879bb-x7zl7\" (UID: \"4e6066cd-42c1-4c15-b699-a0a438779f53\") " pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" Jul 06 21:44:02.631330 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.631261 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e6066cd-42c1-4c15-b699-a0a438779f53-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-563cb-predictor-59bfb879bb-x7zl7\" (UID: \"4e6066cd-42c1-4c15-b699-a0a438779f53\") " pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" Jul 06 21:44:02.631330 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.631281 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-287qf\" (UniqueName: \"kubernetes.io/projected/76c95e95-e959-46eb-b3b6-aa7f979d615f-kube-api-access-287qf\") pod \"error-404-isvc-563cb-predictor-85497489cf-k8hg5\" (UID: \"76c95e95-e959-46eb-b3b6-aa7f979d615f\") " pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" Jul 06 21:44:02.631330 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:44:02.631293 2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-563cb-predictor-serving-cert: secret "success-200-isvc-563cb-predictor-serving-cert" not found Jul 06 21:44:02.631504 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.631299 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnrd6\" (UniqueName: \"kubernetes.io/projected/4e6066cd-42c1-4c15-b699-a0a438779f53-kube-api-access-bnrd6\") pod \"success-200-isvc-563cb-predictor-59bfb879bb-x7zl7\" (UID: \"4e6066cd-42c1-4c15-b699-a0a438779f53\") " pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" Jul 06 21:44:02.631504 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:44:02.631377 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6066cd-42c1-4c15-b699-a0a438779f53-proxy-tls podName:4e6066cd-42c1-4c15-b699-a0a438779f53 nodeName:}" failed. No retries permitted until 2026-07-06 21:44:03.131352902 +0000 UTC m=+1867.894265898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4e6066cd-42c1-4c15-b699-a0a438779f53-proxy-tls") pod "success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" (UID: "4e6066cd-42c1-4c15-b699-a0a438779f53") : secret "success-200-isvc-563cb-predictor-serving-cert" not found Jul 06 21:44:02.631589 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.631491 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76c95e95-e959-46eb-b3b6-aa7f979d615f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-563cb-predictor-85497489cf-k8hg5\" (UID: \"76c95e95-e959-46eb-b3b6-aa7f979d615f\") " pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" Jul 06 21:44:02.631896 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.631878 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e6066cd-42c1-4c15-b699-a0a438779f53-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-563cb-predictor-59bfb879bb-x7zl7\" (UID: \"4e6066cd-42c1-4c15-b699-a0a438779f53\") " pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" Jul 06 21:44:02.643208 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.643187 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnrd6\" (UniqueName: \"kubernetes.io/projected/4e6066cd-42c1-4c15-b699-a0a438779f53-kube-api-access-bnrd6\") pod \"success-200-isvc-563cb-predictor-59bfb879bb-x7zl7\" (UID: \"4e6066cd-42c1-4c15-b699-a0a438779f53\") " pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" Jul 06 21:44:02.732520 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.732487 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-287qf\" (UniqueName: \"kubernetes.io/projected/76c95e95-e959-46eb-b3b6-aa7f979d615f-kube-api-access-287qf\") pod \"error-404-isvc-563cb-predictor-85497489cf-k8hg5\" (UID: \"76c95e95-e959-46eb-b3b6-aa7f979d615f\") " pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" Jul 06 21:44:02.732520 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.732529 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76c95e95-e959-46eb-b3b6-aa7f979d615f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-563cb-predictor-85497489cf-k8hg5\" (UID: \"76c95e95-e959-46eb-b3b6-aa7f979d615f\") " pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" Jul 06 21:44:02.732764 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.732631 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76c95e95-e959-46eb-b3b6-aa7f979d615f-proxy-tls\") pod \"error-404-isvc-563cb-predictor-85497489cf-k8hg5\" (UID: \"76c95e95-e959-46eb-b3b6-aa7f979d615f\") " pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" Jul 06 21:44:02.732764 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:44:02.732717 2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-563cb-predictor-serving-cert: secret "error-404-isvc-563cb-predictor-serving-cert" not found Jul 06 21:44:02.732764 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:44:02.732763 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c95e95-e959-46eb-b3b6-aa7f979d615f-proxy-tls podName:76c95e95-e959-46eb-b3b6-aa7f979d615f nodeName:}" failed. No retries permitted until 2026-07-06 21:44:03.232748755 +0000 UTC m=+1867.995661742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/76c95e95-e959-46eb-b3b6-aa7f979d615f-proxy-tls") pod "error-404-isvc-563cb-predictor-85497489cf-k8hg5" (UID: "76c95e95-e959-46eb-b3b6-aa7f979d615f") : secret "error-404-isvc-563cb-predictor-serving-cert" not found Jul 06 21:44:02.733230 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.733209 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76c95e95-e959-46eb-b3b6-aa7f979d615f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-563cb-predictor-85497489cf-k8hg5\" (UID: \"76c95e95-e959-46eb-b3b6-aa7f979d615f\") " pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" Jul 06 21:44:02.741938 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.741911 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-287qf\" (UniqueName: \"kubernetes.io/projected/76c95e95-e959-46eb-b3b6-aa7f979d615f-kube-api-access-287qf\") pod \"error-404-isvc-563cb-predictor-85497489cf-k8hg5\" (UID: \"76c95e95-e959-46eb-b3b6-aa7f979d615f\") " pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" Jul 06 21:44:02.886878 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.886797 2581 generic.go:356] "Generic (PLEG): container finished" podID="1f0d4bf0-2bec-413c-98d1-e3a5f264ab26" containerID="37308f37bc0765f228200fd64fa35ddb1ca252dc917ccd6bae4af0a9e67af3ea" exitCode=2 Jul 06 21:44:02.887024 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.886870 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" event={"ID":"1f0d4bf0-2bec-413c-98d1-e3a5f264ab26","Type":"ContainerDied","Data":"37308f37bc0765f228200fd64fa35ddb1ca252dc917ccd6bae4af0a9e67af3ea"} Jul 06 21:44:02.888636 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.888606 2581 generic.go:356] "Generic (PLEG): container finished" podID="639574ec-8467-4bb3-9203-6168444f8331" containerID="3ee7c6bc977a21cae874272ac6098360909dbe7dd44f80acf94fc2db1fb3fdf7" exitCode=2 Jul 06 21:44:02.888743 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:02.888677 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" event={"ID":"639574ec-8467-4bb3-9203-6168444f8331","Type":"ContainerDied","Data":"3ee7c6bc977a21cae874272ac6098360909dbe7dd44f80acf94fc2db1fb3fdf7"} Jul 06 21:44:03.136976 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:03.136886 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e6066cd-42c1-4c15-b699-a0a438779f53-proxy-tls\") pod \"success-200-isvc-563cb-predictor-59bfb879bb-x7zl7\" (UID: \"4e6066cd-42c1-4c15-b699-a0a438779f53\") " pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" Jul 06 21:44:03.139300 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:03.139272 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e6066cd-42c1-4c15-b699-a0a438779f53-proxy-tls\") pod \"success-200-isvc-563cb-predictor-59bfb879bb-x7zl7\" (UID: \"4e6066cd-42c1-4c15-b699-a0a438779f53\") " pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" Jul 06 21:44:03.237524 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:03.237486 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76c95e95-e959-46eb-b3b6-aa7f979d615f-proxy-tls\") pod \"error-404-isvc-563cb-predictor-85497489cf-k8hg5\" (UID: \"76c95e95-e959-46eb-b3b6-aa7f979d615f\") " pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" Jul 06 21:44:03.239827 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:03.239797 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76c95e95-e959-46eb-b3b6-aa7f979d615f-proxy-tls\") pod \"error-404-isvc-563cb-predictor-85497489cf-k8hg5\" (UID: \"76c95e95-e959-46eb-b3b6-aa7f979d615f\") " pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" Jul 06 21:44:03.291403 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:03.291365 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" Jul 06 21:44:03.390754 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:03.390674 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" Jul 06 21:44:03.428720 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:03.428594 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7"] Jul 06 21:44:03.431420 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:44:03.431390 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6066cd_42c1_4c15_b699_a0a438779f53.slice/crio-72007a24b31e2728389462221cab956f2748b849b2da69b815dadfa42a4960b0 WatchSource:0}: Error finding container 72007a24b31e2728389462221cab956f2748b849b2da69b815dadfa42a4960b0: Status 404 returned error can't find the container with id 72007a24b31e2728389462221cab956f2748b849b2da69b815dadfa42a4960b0 Jul 06 21:44:03.539562 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:03.539522 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5"] Jul 06 21:44:03.541586 ip-10-0-140-54 kubenswrapper[2581]: W0706 21:44:03.541565 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c95e95_e959_46eb_b3b6_aa7f979d615f.slice/crio-d4b8da1efb3a99e8941f0c513c67efeb4148d22a39ebffd8d844ba92205c1839 WatchSource:0}: Error finding container d4b8da1efb3a99e8941f0c513c67efeb4148d22a39ebffd8d844ba92205c1839: Status 404 returned error can't find the container with id d4b8da1efb3a99e8941f0c513c67efeb4148d22a39ebffd8d844ba92205c1839 Jul 06 21:44:03.901275 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:03.901235 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" event={"ID":"76c95e95-e959-46eb-b3b6-aa7f979d615f","Type":"ContainerStarted","Data":"ddb5b58a34d44b06c8527dd35d16b64a16bebb171ffaf98b2db38ff26d633c36"} Jul 06 21:44:03.901496 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:03.901353 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" event={"ID":"76c95e95-e959-46eb-b3b6-aa7f979d615f","Type":"ContainerStarted","Data":"186816f289ef6f716933c2a66cf3803a7964270e0c2bf4459a23fdcf81d0619c"} Jul 06 21:44:03.901496 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:03.901391 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" Jul 06 21:44:03.901496 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:03.901419 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" Jul 06 21:44:03.901496 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:03.901443 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" event={"ID":"76c95e95-e959-46eb-b3b6-aa7f979d615f","Type":"ContainerStarted","Data":"d4b8da1efb3a99e8941f0c513c67efeb4148d22a39ebffd8d844ba92205c1839"} Jul 06 21:44:03.902445 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:03.902413 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" podUID="76c95e95-e959-46eb-b3b6-aa7f979d615f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused" Jul 06 21:44:03.903148 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:03.903122 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" event={"ID":"4e6066cd-42c1-4c15-b699-a0a438779f53","Type":"ContainerStarted","Data":"d2be01e37fe759c38b1780d684066643ceb16946e1802f1105bd00abbdfdcacb"} Jul 06 21:44:03.903222 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:03.903180 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" event={"ID":"4e6066cd-42c1-4c15-b699-a0a438779f53","Type":"ContainerStarted","Data":"9aaad406bccfec01539c0415737c7c938718eb41f61d03f46c3a81c631a8fab5"} Jul 06 21:44:03.903222 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:03.903211 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" event={"ID":"4e6066cd-42c1-4c15-b699-a0a438779f53","Type":"ContainerStarted","Data":"72007a24b31e2728389462221cab956f2748b849b2da69b815dadfa42a4960b0"} Jul 06 21:44:03.903292 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:03.903246 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" Jul 06 21:44:03.903292 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:03.903276 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" Jul 06 21:44:03.904203 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:03.904181 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" podUID="4e6066cd-42c1-4c15-b699-a0a438779f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Jul 06 21:44:03.919690 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:03.919604 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" podStartSLOduration=1.9195889099999999 podStartE2EDuration="1.91958891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:44:03.917401404 +0000 UTC m=+1868.680314411" watchObservedRunningTime="2026-07-06 21:44:03.91958891 +0000 UTC m=+1868.682501916" Jul 06 21:44:03.932564 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:03.932520 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" podStartSLOduration=1.932504579 podStartE2EDuration="1.932504579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:44:03.932181558 +0000 UTC m=+1868.695094565" watchObservedRunningTime="2026-07-06 21:44:03.932504579 +0000 UTC m=+1868.695417584" Jul 06 21:44:04.583804 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:04.583758 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" podUID="639574ec-8467-4bb3-9203-6168444f8331" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused" Jul 06 21:44:04.584263 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:04.583764 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" podUID="1f0d4bf0-2bec-413c-98d1-e3a5f264ab26" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused" Jul 06 21:44:04.908068 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:04.908023 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" podUID="4e6066cd-42c1-4c15-b699-a0a438779f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Jul 06 21:44:04.908263 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:04.908031 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" podUID="76c95e95-e959-46eb-b3b6-aa7f979d615f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused" Jul 06 21:44:06.224764 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.224697 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" Jul 06 21:44:06.370259 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.370233 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f0d4bf0-2bec-413c-98d1-e3a5f264ab26-proxy-tls\") pod \"1f0d4bf0-2bec-413c-98d1-e3a5f264ab26\" (UID: \"1f0d4bf0-2bec-413c-98d1-e3a5f264ab26\") " Jul 06 21:44:06.370384 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.370278 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r64vg\" (UniqueName: \"kubernetes.io/projected/1f0d4bf0-2bec-413c-98d1-e3a5f264ab26-kube-api-access-r64vg\") pod \"1f0d4bf0-2bec-413c-98d1-e3a5f264ab26\" (UID: \"1f0d4bf0-2bec-413c-98d1-e3a5f264ab26\") " Jul 06 21:44:06.370384 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.370312 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f0d4bf0-2bec-413c-98d1-e3a5f264ab26-kube-rbac-proxy-sar-config\") pod \"1f0d4bf0-2bec-413c-98d1-e3a5f264ab26\" (UID: \"1f0d4bf0-2bec-413c-98d1-e3a5f264ab26\") " Jul 06 21:44:06.370751 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.370721 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f0d4bf0-2bec-413c-98d1-e3a5f264ab26-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1f0d4bf0-2bec-413c-98d1-e3a5f264ab26" (UID: "1f0d4bf0-2bec-413c-98d1-e3a5f264ab26"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:44:06.372432 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.372400 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0d4bf0-2bec-413c-98d1-e3a5f264ab26-kube-api-access-r64vg" (OuterVolumeSpecName: "kube-api-access-r64vg") pod "1f0d4bf0-2bec-413c-98d1-e3a5f264ab26" (UID: "1f0d4bf0-2bec-413c-98d1-e3a5f264ab26"). InnerVolumeSpecName "kube-api-access-r64vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:44:06.372530 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.372465 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f0d4bf0-2bec-413c-98d1-e3a5f264ab26-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1f0d4bf0-2bec-413c-98d1-e3a5f264ab26" (UID: "1f0d4bf0-2bec-413c-98d1-e3a5f264ab26"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:44:06.381983 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.381963 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" Jul 06 21:44:06.471756 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.471714 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6jnl\" (UniqueName: \"kubernetes.io/projected/639574ec-8467-4bb3-9203-6168444f8331-kube-api-access-m6jnl\") pod \"639574ec-8467-4bb3-9203-6168444f8331\" (UID: \"639574ec-8467-4bb3-9203-6168444f8331\") " Jul 06 21:44:06.471927 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.471823 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/639574ec-8467-4bb3-9203-6168444f8331-proxy-tls\") pod \"639574ec-8467-4bb3-9203-6168444f8331\" (UID: \"639574ec-8467-4bb3-9203-6168444f8331\") " Jul 06 21:44:06.471927 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.471894 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/639574ec-8467-4bb3-9203-6168444f8331-kube-rbac-proxy-sar-config\") pod \"639574ec-8467-4bb3-9203-6168444f8331\" (UID: \"639574ec-8467-4bb3-9203-6168444f8331\") " Jul 06 21:44:06.472082 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.472068 2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f0d4bf0-2bec-413c-98d1-e3a5f264ab26-proxy-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:44:06.472129 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.472088 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r64vg\" (UniqueName: \"kubernetes.io/projected/1f0d4bf0-2bec-413c-98d1-e3a5f264ab26-kube-api-access-r64vg\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:44:06.472129 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.472102 2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f0d4bf0-2bec-413c-98d1-e3a5f264ab26-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:44:06.472231 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.472205 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/639574ec-8467-4bb3-9203-6168444f8331-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "639574ec-8467-4bb3-9203-6168444f8331" (UID: "639574ec-8467-4bb3-9203-6168444f8331"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:44:06.473896 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.473874 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/639574ec-8467-4bb3-9203-6168444f8331-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "639574ec-8467-4bb3-9203-6168444f8331" (UID: "639574ec-8467-4bb3-9203-6168444f8331"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:44:06.474001 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.473908 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639574ec-8467-4bb3-9203-6168444f8331-kube-api-access-m6jnl" (OuterVolumeSpecName: "kube-api-access-m6jnl") pod "639574ec-8467-4bb3-9203-6168444f8331" (UID: "639574ec-8467-4bb3-9203-6168444f8331"). InnerVolumeSpecName "kube-api-access-m6jnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:44:06.573030 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.572946 2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/639574ec-8467-4bb3-9203-6168444f8331-proxy-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:44:06.573030 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.572976 2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/639574ec-8467-4bb3-9203-6168444f8331-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:44:06.573030 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.572985 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6jnl\" (UniqueName: \"kubernetes.io/projected/639574ec-8467-4bb3-9203-6168444f8331-kube-api-access-m6jnl\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:44:06.918032 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.917999 2581 generic.go:356] "Generic (PLEG): container finished" podID="639574ec-8467-4bb3-9203-6168444f8331" containerID="864c1e72455b16155eba370a7746c2c72f59f25c410be4a102a8178b39486273" exitCode=0 Jul 06 21:44:06.918183 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.918052 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" Jul 06 21:44:06.918183 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.918092 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" event={"ID":"639574ec-8467-4bb3-9203-6168444f8331","Type":"ContainerDied","Data":"864c1e72455b16155eba370a7746c2c72f59f25c410be4a102a8178b39486273"} Jul 06 21:44:06.918183 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.918155 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8" event={"ID":"639574ec-8467-4bb3-9203-6168444f8331","Type":"ContainerDied","Data":"fbbb754984da16696f843aeb723cc2521a190ab12a2c604c59efbef6141f6772"} Jul 06 21:44:06.918365 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.918193 2581 scope.go:117] "RemoveContainer" containerID="3ee7c6bc977a21cae874272ac6098360909dbe7dd44f80acf94fc2db1fb3fdf7" Jul 06 21:44:06.919720 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.919694 2581 generic.go:356] "Generic (PLEG): container finished" podID="1f0d4bf0-2bec-413c-98d1-e3a5f264ab26" containerID="e82962fbbe527837b1f45179531d3ad3008a9765ab7c92fd405b6b1e779690e5" exitCode=0 Jul 06 21:44:06.919843 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.919740 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" event={"ID":"1f0d4bf0-2bec-413c-98d1-e3a5f264ab26","Type":"ContainerDied","Data":"e82962fbbe527837b1f45179531d3ad3008a9765ab7c92fd405b6b1e779690e5"} Jul 06 21:44:06.919843 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.919749 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" Jul 06 21:44:06.919843 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.919790 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd" event={"ID":"1f0d4bf0-2bec-413c-98d1-e3a5f264ab26","Type":"ContainerDied","Data":"fd73a8ff07434ee343b1d5608aa0973af041fbade5ec0b11f1afebaacc774fff"} Jul 06 21:44:06.926823 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.926808 2581 scope.go:117] "RemoveContainer" containerID="864c1e72455b16155eba370a7746c2c72f59f25c410be4a102a8178b39486273" Jul 06 21:44:06.934952 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.934934 2581 scope.go:117] "RemoveContainer" containerID="3ee7c6bc977a21cae874272ac6098360909dbe7dd44f80acf94fc2db1fb3fdf7" Jul 06 21:44:06.935194 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:44:06.935176 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee7c6bc977a21cae874272ac6098360909dbe7dd44f80acf94fc2db1fb3fdf7\": container with ID starting with 3ee7c6bc977a21cae874272ac6098360909dbe7dd44f80acf94fc2db1fb3fdf7 not found: ID does not exist" containerID="3ee7c6bc977a21cae874272ac6098360909dbe7dd44f80acf94fc2db1fb3fdf7" Jul 06 21:44:06.935244 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.935204 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee7c6bc977a21cae874272ac6098360909dbe7dd44f80acf94fc2db1fb3fdf7"} err="failed to get container status \"3ee7c6bc977a21cae874272ac6098360909dbe7dd44f80acf94fc2db1fb3fdf7\": rpc error: code = NotFound desc = could not find container \"3ee7c6bc977a21cae874272ac6098360909dbe7dd44f80acf94fc2db1fb3fdf7\": container with ID starting with 3ee7c6bc977a21cae874272ac6098360909dbe7dd44f80acf94fc2db1fb3fdf7 not found: ID does not exist" Jul 06 21:44:06.935244 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.935222 2581 scope.go:117] "RemoveContainer" containerID="864c1e72455b16155eba370a7746c2c72f59f25c410be4a102a8178b39486273" Jul 06 21:44:06.935457 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:44:06.935441 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"864c1e72455b16155eba370a7746c2c72f59f25c410be4a102a8178b39486273\": container with ID starting with 864c1e72455b16155eba370a7746c2c72f59f25c410be4a102a8178b39486273 not found: ID does not exist" containerID="864c1e72455b16155eba370a7746c2c72f59f25c410be4a102a8178b39486273" Jul 06 21:44:06.935495 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.935462 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"864c1e72455b16155eba370a7746c2c72f59f25c410be4a102a8178b39486273"} err="failed to get container status \"864c1e72455b16155eba370a7746c2c72f59f25c410be4a102a8178b39486273\": rpc error: code = NotFound desc = could not find container \"864c1e72455b16155eba370a7746c2c72f59f25c410be4a102a8178b39486273\": container with ID starting with 864c1e72455b16155eba370a7746c2c72f59f25c410be4a102a8178b39486273 not found: ID does not exist" Jul 06 21:44:06.935495 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.935478 2581 scope.go:117] "RemoveContainer" containerID="37308f37bc0765f228200fd64fa35ddb1ca252dc917ccd6bae4af0a9e67af3ea" Jul 06 21:44:06.942911 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.942893 2581 scope.go:117] "RemoveContainer" containerID="e82962fbbe527837b1f45179531d3ad3008a9765ab7c92fd405b6b1e779690e5" Jul 06 21:44:06.943796 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.943778 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8"] Jul 06 21:44:06.949358 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.949317 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-974ed-predictor-677df94fc9-c69t8"] Jul 06 21:44:06.951967 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.951949 2581 scope.go:117] "RemoveContainer" containerID="37308f37bc0765f228200fd64fa35ddb1ca252dc917ccd6bae4af0a9e67af3ea" Jul 06 21:44:06.952245 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:44:06.952221 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37308f37bc0765f228200fd64fa35ddb1ca252dc917ccd6bae4af0a9e67af3ea\": container with ID starting with 37308f37bc0765f228200fd64fa35ddb1ca252dc917ccd6bae4af0a9e67af3ea not found: ID does not exist" containerID="37308f37bc0765f228200fd64fa35ddb1ca252dc917ccd6bae4af0a9e67af3ea" Jul 06 21:44:06.952328 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.952251 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37308f37bc0765f228200fd64fa35ddb1ca252dc917ccd6bae4af0a9e67af3ea"} err="failed to get container status \"37308f37bc0765f228200fd64fa35ddb1ca252dc917ccd6bae4af0a9e67af3ea\": rpc error: code = NotFound desc = could not find container \"37308f37bc0765f228200fd64fa35ddb1ca252dc917ccd6bae4af0a9e67af3ea\": container with ID starting with 37308f37bc0765f228200fd64fa35ddb1ca252dc917ccd6bae4af0a9e67af3ea not found: ID does not exist" Jul 06 21:44:06.952328 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.952273 2581 scope.go:117] "RemoveContainer" containerID="e82962fbbe527837b1f45179531d3ad3008a9765ab7c92fd405b6b1e779690e5" Jul 06 21:44:06.952582 ip-10-0-140-54 kubenswrapper[2581]: E0706 21:44:06.952562 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e82962fbbe527837b1f45179531d3ad3008a9765ab7c92fd405b6b1e779690e5\": container with ID starting with e82962fbbe527837b1f45179531d3ad3008a9765ab7c92fd405b6b1e779690e5 not found: ID does not exist" containerID="e82962fbbe527837b1f45179531d3ad3008a9765ab7c92fd405b6b1e779690e5" Jul 06 21:44:06.952664 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.952587 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e82962fbbe527837b1f45179531d3ad3008a9765ab7c92fd405b6b1e779690e5"} err="failed to get container status \"e82962fbbe527837b1f45179531d3ad3008a9765ab7c92fd405b6b1e779690e5\": rpc error: code = NotFound desc = could not find container \"e82962fbbe527837b1f45179531d3ad3008a9765ab7c92fd405b6b1e779690e5\": container with ID starting with e82962fbbe527837b1f45179531d3ad3008a9765ab7c92fd405b6b1e779690e5 not found: ID does not exist" Jul 06 21:44:06.961717 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.961689 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd"] Jul 06 21:44:06.965200 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:06.965171 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-974ed-predictor-7bfd566fb6-zs7xd"] Jul 06 21:44:07.785985 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:07.785952 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f0d4bf0-2bec-413c-98d1-e3a5f264ab26" path="/var/lib/kubelet/pods/1f0d4bf0-2bec-413c-98d1-e3a5f264ab26/volumes" Jul 06 21:44:07.786397 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:07.786383 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="639574ec-8467-4bb3-9203-6168444f8331" path="/var/lib/kubelet/pods/639574ec-8467-4bb3-9203-6168444f8331/volumes" Jul 06 21:44:08.840970 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:08.840930 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" podUID="5913d04b-be39-4226-8825-68074a5e74ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Jul 06 21:44:09.846270 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:09.846234 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" podUID="f20475b7-1ad4-43b0-b506-e72e28b106f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Jul 06 21:44:09.912724 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:09.912539 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" Jul 06 21:44:09.913027 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:09.912993 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" podUID="4e6066cd-42c1-4c15-b699-a0a438779f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Jul 06 21:44:09.913670 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:09.913648 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" Jul 06 21:44:09.914067 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:09.914046 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" podUID="76c95e95-e959-46eb-b3b6-aa7f979d615f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused" Jul 06 21:44:18.841278 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:18.841172 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" podUID="5913d04b-be39-4226-8825-68074a5e74ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Jul 06 21:44:19.845912 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:19.845869 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" podUID="f20475b7-1ad4-43b0-b506-e72e28b106f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Jul 06 21:44:19.913820 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:19.913777 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" podUID="4e6066cd-42c1-4c15-b699-a0a438779f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Jul 06 21:44:19.914038 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:19.914009 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" podUID="76c95e95-e959-46eb-b3b6-aa7f979d615f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused" Jul 06 21:44:28.840388 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:28.840319 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" podUID="5913d04b-be39-4226-8825-68074a5e74ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Jul 06 21:44:29.845835 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:29.845792 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" podUID="f20475b7-1ad4-43b0-b506-e72e28b106f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Jul 06 21:44:29.913751 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:29.913713 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" podUID="4e6066cd-42c1-4c15-b699-a0a438779f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Jul 06 21:44:29.914008 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:29.913983 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" podUID="76c95e95-e959-46eb-b3b6-aa7f979d615f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused" Jul 06 21:44:38.840759 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:38.840723 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" podUID="5913d04b-be39-4226-8825-68074a5e74ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Jul 06 21:44:39.845576 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:39.845531 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" podUID="f20475b7-1ad4-43b0-b506-e72e28b106f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Jul 06 21:44:39.913635 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:39.913589 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" podUID="4e6066cd-42c1-4c15-b699-a0a438779f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Jul 06 21:44:39.914027 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:39.914002 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" podUID="76c95e95-e959-46eb-b3b6-aa7f979d615f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused" Jul 06 21:44:48.841532 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:48.841499 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" Jul 06 21:44:49.846487 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:49.846453 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" Jul 06 21:44:49.913138 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:49.913095 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" podUID="4e6066cd-42c1-4c15-b699-a0a438779f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Jul 06 21:44:49.914309 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:49.914281 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" podUID="76c95e95-e959-46eb-b3b6-aa7f979d615f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused" Jul 06 21:44:59.914502 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:59.914468 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" Jul 06 21:44:59.914953 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:44:59.914549 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" Jul 06 21:53:27.220290 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:27.220196 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7"] Jul 06 21:53:27.222802 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:27.220495 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" podUID="4e6066cd-42c1-4c15-b699-a0a438779f53" containerName="kube-rbac-proxy" containerID="cri-o://d2be01e37fe759c38b1780d684066643ceb16946e1802f1105bd00abbdfdcacb" gracePeriod=30 Jul 06 21:53:27.222802 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:27.220484 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" podUID="4e6066cd-42c1-4c15-b699-a0a438779f53" containerName="kserve-container" containerID="cri-o://9aaad406bccfec01539c0415737c7c938718eb41f61d03f46c3a81c631a8fab5" gracePeriod=30 Jul 06 21:53:27.253672 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:27.253646 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5"] Jul 06 21:53:27.253895 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:27.253859 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" podUID="76c95e95-e959-46eb-b3b6-aa7f979d615f" containerName="kserve-container" containerID="cri-o://186816f289ef6f716933c2a66cf3803a7964270e0c2bf4459a23fdcf81d0619c" gracePeriod=30 Jul 06 21:53:27.253953 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:27.253885 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" podUID="76c95e95-e959-46eb-b3b6-aa7f979d615f" containerName="kube-rbac-proxy" containerID="cri-o://ddb5b58a34d44b06c8527dd35d16b64a16bebb171ffaf98b2db38ff26d633c36" gracePeriod=30 Jul 06 21:53:28.259656 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:28.259619 2581 generic.go:356] "Generic (PLEG): container finished" podID="76c95e95-e959-46eb-b3b6-aa7f979d615f" containerID="ddb5b58a34d44b06c8527dd35d16b64a16bebb171ffaf98b2db38ff26d633c36" exitCode=2 Jul 06 21:53:28.260071 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:28.259687 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" event={"ID":"76c95e95-e959-46eb-b3b6-aa7f979d615f","Type":"ContainerDied","Data":"ddb5b58a34d44b06c8527dd35d16b64a16bebb171ffaf98b2db38ff26d633c36"} Jul 06 21:53:28.261246 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:28.261226 2581 generic.go:356] "Generic (PLEG): container finished" podID="4e6066cd-42c1-4c15-b699-a0a438779f53" containerID="d2be01e37fe759c38b1780d684066643ceb16946e1802f1105bd00abbdfdcacb" exitCode=2 Jul 06 21:53:28.261367 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:28.261286 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" event={"ID":"4e6066cd-42c1-4c15-b699-a0a438779f53","Type":"ContainerDied","Data":"d2be01e37fe759c38b1780d684066643ceb16946e1802f1105bd00abbdfdcacb"} Jul 06 21:53:29.908597 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:29.908556 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" podUID="76c95e95-e959-46eb-b3b6-aa7f979d615f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused" Jul 06 21:53:29.909029 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:29.908557 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" podUID="4e6066cd-42c1-4c15-b699-a0a438779f53" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused" Jul 06 21:53:29.913491 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:29.913461 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" podUID="4e6066cd-42c1-4c15-b699-a0a438779f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Jul 06 21:53:29.914685 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:29.914656 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" podUID="76c95e95-e959-46eb-b3b6-aa7f979d615f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused" Jul 06 21:53:30.272105 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.272073 2581 generic.go:356] "Generic (PLEG): container finished" podID="4e6066cd-42c1-4c15-b699-a0a438779f53" containerID="9aaad406bccfec01539c0415737c7c938718eb41f61d03f46c3a81c631a8fab5" exitCode=0 Jul 06 21:53:30.272204 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.272164 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" event={"ID":"4e6066cd-42c1-4c15-b699-a0a438779f53","Type":"ContainerDied","Data":"9aaad406bccfec01539c0415737c7c938718eb41f61d03f46c3a81c631a8fab5"} Jul 06 21:53:30.273903 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.273880 2581 generic.go:356] "Generic (PLEG): container finished" podID="76c95e95-e959-46eb-b3b6-aa7f979d615f" containerID="186816f289ef6f716933c2a66cf3803a7964270e0c2bf4459a23fdcf81d0619c" exitCode=0 Jul 06 21:53:30.274002 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.273926 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" event={"ID":"76c95e95-e959-46eb-b3b6-aa7f979d615f","Type":"ContainerDied","Data":"186816f289ef6f716933c2a66cf3803a7964270e0c2bf4459a23fdcf81d0619c"} Jul 06 21:53:30.382536 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.382508 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" Jul 06 21:53:30.409212 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.409176 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" Jul 06 21:53:30.490455 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.490361 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76c95e95-e959-46eb-b3b6-aa7f979d615f-proxy-tls\") pod \"76c95e95-e959-46eb-b3b6-aa7f979d615f\" (UID: \"76c95e95-e959-46eb-b3b6-aa7f979d615f\") " Jul 06 21:53:30.490455 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.490402 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnrd6\" (UniqueName: \"kubernetes.io/projected/4e6066cd-42c1-4c15-b699-a0a438779f53-kube-api-access-bnrd6\") pod \"4e6066cd-42c1-4c15-b699-a0a438779f53\" (UID: \"4e6066cd-42c1-4c15-b699-a0a438779f53\") " Jul 06 21:53:30.490455 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.490422 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76c95e95-e959-46eb-b3b6-aa7f979d615f-kube-rbac-proxy-sar-config\") pod \"76c95e95-e959-46eb-b3b6-aa7f979d615f\" (UID: \"76c95e95-e959-46eb-b3b6-aa7f979d615f\") " Jul 06 21:53:30.490455 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.490446 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-287qf\" (UniqueName: \"kubernetes.io/projected/76c95e95-e959-46eb-b3b6-aa7f979d615f-kube-api-access-287qf\") pod \"76c95e95-e959-46eb-b3b6-aa7f979d615f\" (UID: \"76c95e95-e959-46eb-b3b6-aa7f979d615f\") " Jul 06 21:53:30.490796 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.490478 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e6066cd-42c1-4c15-b699-a0a438779f53-kube-rbac-proxy-sar-config\") pod \"4e6066cd-42c1-4c15-b699-a0a438779f53\" (UID: \"4e6066cd-42c1-4c15-b699-a0a438779f53\") " Jul 06 21:53:30.490796 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.490508 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e6066cd-42c1-4c15-b699-a0a438779f53-proxy-tls\") pod \"4e6066cd-42c1-4c15-b699-a0a438779f53\" (UID: \"4e6066cd-42c1-4c15-b699-a0a438779f53\") " Jul 06 21:53:30.490905 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.490790 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c95e95-e959-46eb-b3b6-aa7f979d615f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "76c95e95-e959-46eb-b3b6-aa7f979d615f" (UID: "76c95e95-e959-46eb-b3b6-aa7f979d615f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:53:30.490905 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.490861 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6066cd-42c1-4c15-b699-a0a438779f53-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4e6066cd-42c1-4c15-b699-a0a438779f53" (UID: "4e6066cd-42c1-4c15-b699-a0a438779f53"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:53:30.492625 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.492596 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e6066cd-42c1-4c15-b699-a0a438779f53-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4e6066cd-42c1-4c15-b699-a0a438779f53" (UID: "4e6066cd-42c1-4c15-b699-a0a438779f53"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:53:30.492625 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.492618 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c95e95-e959-46eb-b3b6-aa7f979d615f-kube-api-access-287qf" (OuterVolumeSpecName: "kube-api-access-287qf") pod "76c95e95-e959-46eb-b3b6-aa7f979d615f" (UID: "76c95e95-e959-46eb-b3b6-aa7f979d615f"). InnerVolumeSpecName "kube-api-access-287qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:53:30.492781 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.492651 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76c95e95-e959-46eb-b3b6-aa7f979d615f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "76c95e95-e959-46eb-b3b6-aa7f979d615f" (UID: "76c95e95-e959-46eb-b3b6-aa7f979d615f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:53:30.492781 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.492671 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6066cd-42c1-4c15-b699-a0a438779f53-kube-api-access-bnrd6" (OuterVolumeSpecName: "kube-api-access-bnrd6") pod "4e6066cd-42c1-4c15-b699-a0a438779f53" (UID: "4e6066cd-42c1-4c15-b699-a0a438779f53"). InnerVolumeSpecName "kube-api-access-bnrd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:53:30.591614 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.591572 2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76c95e95-e959-46eb-b3b6-aa7f979d615f-proxy-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:53:30.591614 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.591610 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnrd6\" (UniqueName: \"kubernetes.io/projected/4e6066cd-42c1-4c15-b699-a0a438779f53-kube-api-access-bnrd6\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:53:30.591820 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.591623 2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76c95e95-e959-46eb-b3b6-aa7f979d615f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:53:30.591820 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.591636 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-287qf\" (UniqueName: \"kubernetes.io/projected/76c95e95-e959-46eb-b3b6-aa7f979d615f-kube-api-access-287qf\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:53:30.591820 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.591650 2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e6066cd-42c1-4c15-b699-a0a438779f53-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:53:30.591820 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:30.591662 2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e6066cd-42c1-4c15-b699-a0a438779f53-proxy-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 21:53:31.279800 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:31.279770 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" Jul 06 21:53:31.279800 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:31.279783 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5" event={"ID":"76c95e95-e959-46eb-b3b6-aa7f979d615f","Type":"ContainerDied","Data":"d4b8da1efb3a99e8941f0c513c67efeb4148d22a39ebffd8d844ba92205c1839"} Jul 06 21:53:31.280362 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:31.279878 2581 scope.go:117] "RemoveContainer" containerID="ddb5b58a34d44b06c8527dd35d16b64a16bebb171ffaf98b2db38ff26d633c36" Jul 06 21:53:31.281587 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:31.281557 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" event={"ID":"4e6066cd-42c1-4c15-b699-a0a438779f53","Type":"ContainerDied","Data":"72007a24b31e2728389462221cab956f2748b849b2da69b815dadfa42a4960b0"} Jul 06 21:53:31.281587 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:31.281576 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7" Jul 06 21:53:31.289857 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:31.289841 2581 scope.go:117] "RemoveContainer" containerID="186816f289ef6f716933c2a66cf3803a7964270e0c2bf4459a23fdcf81d0619c" Jul 06 21:53:31.297646 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:31.297625 2581 scope.go:117] "RemoveContainer" containerID="d2be01e37fe759c38b1780d684066643ceb16946e1802f1105bd00abbdfdcacb" Jul 06 21:53:31.302684 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:31.302664 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5"] Jul 06 21:53:31.305643 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:31.305618 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-563cb-predictor-85497489cf-k8hg5"] Jul 06 21:53:31.307504 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:31.307489 2581 scope.go:117] "RemoveContainer" containerID="9aaad406bccfec01539c0415737c7c938718eb41f61d03f46c3a81c631a8fab5" Jul 06 21:53:31.316209 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:31.316185 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7"] Jul 06 21:53:31.319788 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:31.319767 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-563cb-predictor-59bfb879bb-x7zl7"] Jul 06 21:53:31.785998 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:31.785957 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e6066cd-42c1-4c15-b699-a0a438779f53" path="/var/lib/kubelet/pods/4e6066cd-42c1-4c15-b699-a0a438779f53/volumes" Jul 06 21:53:31.786503 ip-10-0-140-54 kubenswrapper[2581]: I0706 21:53:31.786483 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76c95e95-e959-46eb-b3b6-aa7f979d615f" path="/var/lib/kubelet/pods/76c95e95-e959-46eb-b3b6-aa7f979d615f/volumes" Jul 06 22:01:21.518156 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:21.518065 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx"] Jul 06 22:01:21.520716 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:21.518383 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" podUID="5913d04b-be39-4226-8825-68074a5e74ea" containerName="kserve-container" containerID="cri-o://865553cc212221dfc0a2512e3fa2e0fc000c515b403bcec127d87c052870b5a8" gracePeriod=30 Jul 06 22:01:21.520716 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:21.518402 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" podUID="5913d04b-be39-4226-8825-68074a5e74ea" containerName="kube-rbac-proxy" containerID="cri-o://5bb1ac559587d481cb73ab3f0d5a69a271e2a8f279501dd332629792bb598bd7" gracePeriod=30 Jul 06 22:01:21.574169 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:21.574130 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj"] Jul 06 22:01:21.574410 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:21.574382 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" podUID="f20475b7-1ad4-43b0-b506-e72e28b106f4" containerName="kserve-container" containerID="cri-o://00047d17bbad2f47c8cfc2ab2a8bf0dab9c2eb5b9ced0b4d0fa30f090baff9d3" gracePeriod=30 Jul 06 22:01:21.574556 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:21.574422 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" podUID="f20475b7-1ad4-43b0-b506-e72e28b106f4" containerName="kube-rbac-proxy" containerID="cri-o://c89ce198514bd851b421357f74efa0677d1b82abed6c95fc7473475e85113278" gracePeriod=30 Jul 06 22:01:22.176720 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:22.176682 2581 generic.go:356] "Generic (PLEG): container finished" podID="f20475b7-1ad4-43b0-b506-e72e28b106f4" containerID="c89ce198514bd851b421357f74efa0677d1b82abed6c95fc7473475e85113278" exitCode=2 Jul 06 22:01:22.176905 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:22.176754 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" event={"ID":"f20475b7-1ad4-43b0-b506-e72e28b106f4","Type":"ContainerDied","Data":"c89ce198514bd851b421357f74efa0677d1b82abed6c95fc7473475e85113278"} Jul 06 22:01:22.178437 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:22.178415 2581 generic.go:356] "Generic (PLEG): container finished" podID="5913d04b-be39-4226-8825-68074a5e74ea" containerID="5bb1ac559587d481cb73ab3f0d5a69a271e2a8f279501dd332629792bb598bd7" exitCode=2 Jul 06 22:01:22.178544 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:22.178482 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" event={"ID":"5913d04b-be39-4226-8825-68074a5e74ea","Type":"ContainerDied","Data":"5bb1ac559587d481cb73ab3f0d5a69a271e2a8f279501dd332629792bb598bd7"} Jul 06 22:01:23.835721 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:23.835685 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" podUID="5913d04b-be39-4226-8825-68074a5e74ea" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused" Jul 06 22:01:24.553181 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:24.553144 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" Jul 06 22:01:24.584368 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:24.584333 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" Jul 06 22:01:24.638975 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:24.638908 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqhjk\" (UniqueName: \"kubernetes.io/projected/f20475b7-1ad4-43b0-b506-e72e28b106f4-kube-api-access-pqhjk\") pod \"f20475b7-1ad4-43b0-b506-e72e28b106f4\" (UID: \"f20475b7-1ad4-43b0-b506-e72e28b106f4\") " Jul 06 22:01:24.638975 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:24.638947 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8qqt\" (UniqueName: \"kubernetes.io/projected/5913d04b-be39-4226-8825-68074a5e74ea-kube-api-access-b8qqt\") pod \"5913d04b-be39-4226-8825-68074a5e74ea\" (UID: \"5913d04b-be39-4226-8825-68074a5e74ea\") " Jul 06 22:01:24.639166 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:24.638990 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5913d04b-be39-4226-8825-68074a5e74ea-kube-rbac-proxy-sar-config\") pod \"5913d04b-be39-4226-8825-68074a5e74ea\" (UID: \"5913d04b-be39-4226-8825-68074a5e74ea\") " Jul 06 22:01:24.639166 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:24.639040 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f20475b7-1ad4-43b0-b506-e72e28b106f4-proxy-tls\") pod \"f20475b7-1ad4-43b0-b506-e72e28b106f4\" (UID: \"f20475b7-1ad4-43b0-b506-e72e28b106f4\") " Jul 06 22:01:24.639166 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:24.639070 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f20475b7-1ad4-43b0-b506-e72e28b106f4-kube-rbac-proxy-sar-config\") pod \"f20475b7-1ad4-43b0-b506-e72e28b106f4\" (UID: \"f20475b7-1ad4-43b0-b506-e72e28b106f4\") " Jul 06 22:01:24.639507 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:24.639481 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f20475b7-1ad4-43b0-b506-e72e28b106f4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f20475b7-1ad4-43b0-b506-e72e28b106f4" (UID: "f20475b7-1ad4-43b0-b506-e72e28b106f4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:01:24.639680 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:24.639654 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5913d04b-be39-4226-8825-68074a5e74ea-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5913d04b-be39-4226-8825-68074a5e74ea" (UID: "5913d04b-be39-4226-8825-68074a5e74ea"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:01:24.640963 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:24.640939 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20475b7-1ad4-43b0-b506-e72e28b106f4-kube-api-access-pqhjk" (OuterVolumeSpecName: "kube-api-access-pqhjk") pod "f20475b7-1ad4-43b0-b506-e72e28b106f4" (UID: "f20475b7-1ad4-43b0-b506-e72e28b106f4"). InnerVolumeSpecName "kube-api-access-pqhjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:01:24.641054 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:24.640976 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f20475b7-1ad4-43b0-b506-e72e28b106f4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f20475b7-1ad4-43b0-b506-e72e28b106f4" (UID: "f20475b7-1ad4-43b0-b506-e72e28b106f4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:01:24.641054 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:24.641022 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5913d04b-be39-4226-8825-68074a5e74ea-kube-api-access-b8qqt" (OuterVolumeSpecName: "kube-api-access-b8qqt") pod "5913d04b-be39-4226-8825-68074a5e74ea" (UID: "5913d04b-be39-4226-8825-68074a5e74ea"). InnerVolumeSpecName "kube-api-access-b8qqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:01:24.740001 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:24.739977 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5913d04b-be39-4226-8825-68074a5e74ea-proxy-tls\") pod \"5913d04b-be39-4226-8825-68074a5e74ea\" (UID: \"5913d04b-be39-4226-8825-68074a5e74ea\") " Jul 06 22:01:24.740163 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:24.740150 2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f20475b7-1ad4-43b0-b506-e72e28b106f4-proxy-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 22:01:24.740216 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:24.740166 2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f20475b7-1ad4-43b0-b506-e72e28b106f4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 22:01:24.740216 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:24.740175 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqhjk\" (UniqueName: \"kubernetes.io/projected/f20475b7-1ad4-43b0-b506-e72e28b106f4-kube-api-access-pqhjk\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 22:01:24.740216 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:24.740184 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8qqt\" (UniqueName: \"kubernetes.io/projected/5913d04b-be39-4226-8825-68074a5e74ea-kube-api-access-b8qqt\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 22:01:24.740216 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:24.740192 2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5913d04b-be39-4226-8825-68074a5e74ea-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 22:01:24.741853 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:24.741833 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5913d04b-be39-4226-8825-68074a5e74ea-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5913d04b-be39-4226-8825-68074a5e74ea" (UID: "5913d04b-be39-4226-8825-68074a5e74ea"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:01:24.840533 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:24.840506 2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5913d04b-be39-4226-8825-68074a5e74ea-proxy-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\"" Jul 06 22:01:25.191329 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.191299 2581 generic.go:356] "Generic (PLEG): container finished" podID="f20475b7-1ad4-43b0-b506-e72e28b106f4" containerID="00047d17bbad2f47c8cfc2ab2a8bf0dab9c2eb5b9ced0b4d0fa30f090baff9d3" exitCode=0 Jul 06 22:01:25.191544 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.191376 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" Jul 06 22:01:25.191544 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.191385 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" event={"ID":"f20475b7-1ad4-43b0-b506-e72e28b106f4","Type":"ContainerDied","Data":"00047d17bbad2f47c8cfc2ab2a8bf0dab9c2eb5b9ced0b4d0fa30f090baff9d3"} Jul 06 22:01:25.191544 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.191456 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj" event={"ID":"f20475b7-1ad4-43b0-b506-e72e28b106f4","Type":"ContainerDied","Data":"6380d58be1ceab0bb7fa3cc62bf69a9fc7aac59b852411cede41f27e1954094e"} Jul 06 22:01:25.191544 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.191486 2581 scope.go:117] "RemoveContainer" containerID="c89ce198514bd851b421357f74efa0677d1b82abed6c95fc7473475e85113278" Jul 06 22:01:25.192948 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.192927 2581 generic.go:356] "Generic (PLEG): container finished" podID="5913d04b-be39-4226-8825-68074a5e74ea" containerID="865553cc212221dfc0a2512e3fa2e0fc000c515b403bcec127d87c052870b5a8" exitCode=0 Jul 06 22:01:25.193039 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.192973 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" Jul 06 22:01:25.193098 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.193032 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" event={"ID":"5913d04b-be39-4226-8825-68074a5e74ea","Type":"ContainerDied","Data":"865553cc212221dfc0a2512e3fa2e0fc000c515b403bcec127d87c052870b5a8"} Jul 06 22:01:25.193135 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.193095 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx" event={"ID":"5913d04b-be39-4226-8825-68074a5e74ea","Type":"ContainerDied","Data":"1dffc05077687776c668b23991ca8c8a3feb7cf93eeaa3127939b61b31e99d33"} Jul 06 22:01:25.201113 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.200986 2581 scope.go:117] "RemoveContainer" containerID="00047d17bbad2f47c8cfc2ab2a8bf0dab9c2eb5b9ced0b4d0fa30f090baff9d3" Jul 06 22:01:25.208685 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.208669 2581 scope.go:117] "RemoveContainer" containerID="c89ce198514bd851b421357f74efa0677d1b82abed6c95fc7473475e85113278" Jul 06 22:01:25.208948 ip-10-0-140-54 kubenswrapper[2581]: E0706 22:01:25.208932 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c89ce198514bd851b421357f74efa0677d1b82abed6c95fc7473475e85113278\": container with ID starting with c89ce198514bd851b421357f74efa0677d1b82abed6c95fc7473475e85113278 not found: ID does not exist" containerID="c89ce198514bd851b421357f74efa0677d1b82abed6c95fc7473475e85113278" Jul 06 22:01:25.208988 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.208957 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c89ce198514bd851b421357f74efa0677d1b82abed6c95fc7473475e85113278"} err="failed to get container status \"c89ce198514bd851b421357f74efa0677d1b82abed6c95fc7473475e85113278\": rpc error: code = NotFound desc = could not find container \"c89ce198514bd851b421357f74efa0677d1b82abed6c95fc7473475e85113278\": container with ID starting with c89ce198514bd851b421357f74efa0677d1b82abed6c95fc7473475e85113278 not found: ID does not exist" Jul 06 22:01:25.208988 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.208974 2581 scope.go:117] "RemoveContainer" containerID="00047d17bbad2f47c8cfc2ab2a8bf0dab9c2eb5b9ced0b4d0fa30f090baff9d3" Jul 06 22:01:25.209181 ip-10-0-140-54 kubenswrapper[2581]: E0706 22:01:25.209157 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00047d17bbad2f47c8cfc2ab2a8bf0dab9c2eb5b9ced0b4d0fa30f090baff9d3\": container with ID starting with 00047d17bbad2f47c8cfc2ab2a8bf0dab9c2eb5b9ced0b4d0fa30f090baff9d3 not found: ID does not exist" containerID="00047d17bbad2f47c8cfc2ab2a8bf0dab9c2eb5b9ced0b4d0fa30f090baff9d3" Jul 06 22:01:25.209220 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.209190 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00047d17bbad2f47c8cfc2ab2a8bf0dab9c2eb5b9ced0b4d0fa30f090baff9d3"} err="failed to get container status \"00047d17bbad2f47c8cfc2ab2a8bf0dab9c2eb5b9ced0b4d0fa30f090baff9d3\": rpc error: code = NotFound desc = could not find container \"00047d17bbad2f47c8cfc2ab2a8bf0dab9c2eb5b9ced0b4d0fa30f090baff9d3\": container with ID starting with 00047d17bbad2f47c8cfc2ab2a8bf0dab9c2eb5b9ced0b4d0fa30f090baff9d3 not found: ID does not exist" Jul 06 22:01:25.209220 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.209206 2581 scope.go:117] "RemoveContainer" containerID="5bb1ac559587d481cb73ab3f0d5a69a271e2a8f279501dd332629792bb598bd7" Jul 06 22:01:25.216592 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.216571 2581 scope.go:117] "RemoveContainer" containerID="865553cc212221dfc0a2512e3fa2e0fc000c515b403bcec127d87c052870b5a8" Jul 06 22:01:25.217872 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.217854 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx"] Jul 06 22:01:25.223503 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.223480 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-da1f6-predictor-6bb4478c84-hc5zx"] Jul 06 22:01:25.224085 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.224062 2581 scope.go:117] "RemoveContainer" containerID="5bb1ac559587d481cb73ab3f0d5a69a271e2a8f279501dd332629792bb598bd7" Jul 06 22:01:25.224334 ip-10-0-140-54 kubenswrapper[2581]: E0706 22:01:25.224317 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bb1ac559587d481cb73ab3f0d5a69a271e2a8f279501dd332629792bb598bd7\": container with ID starting with 5bb1ac559587d481cb73ab3f0d5a69a271e2a8f279501dd332629792bb598bd7 not found: ID does not exist" containerID="5bb1ac559587d481cb73ab3f0d5a69a271e2a8f279501dd332629792bb598bd7" Jul 06 22:01:25.224334 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.224353 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb1ac559587d481cb73ab3f0d5a69a271e2a8f279501dd332629792bb598bd7"} err="failed to get container status \"5bb1ac559587d481cb73ab3f0d5a69a271e2a8f279501dd332629792bb598bd7\": rpc error: code = NotFound desc = could not find container \"5bb1ac559587d481cb73ab3f0d5a69a271e2a8f279501dd332629792bb598bd7\": container with ID starting with 5bb1ac559587d481cb73ab3f0d5a69a271e2a8f279501dd332629792bb598bd7 not found: ID does not exist" Jul 06 22:01:25.224334 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.224370 2581 scope.go:117] "RemoveContainer" containerID="865553cc212221dfc0a2512e3fa2e0fc000c515b403bcec127d87c052870b5a8" Jul 06 22:01:25.224613 ip-10-0-140-54 kubenswrapper[2581]: E0706 22:01:25.224596 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"865553cc212221dfc0a2512e3fa2e0fc000c515b403bcec127d87c052870b5a8\": container with ID starting with 865553cc212221dfc0a2512e3fa2e0fc000c515b403bcec127d87c052870b5a8 not found: ID does not exist" containerID="865553cc212221dfc0a2512e3fa2e0fc000c515b403bcec127d87c052870b5a8" Jul 06 22:01:25.224647 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.224617 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865553cc212221dfc0a2512e3fa2e0fc000c515b403bcec127d87c052870b5a8"} err="failed to get container status \"865553cc212221dfc0a2512e3fa2e0fc000c515b403bcec127d87c052870b5a8\": rpc error: code = NotFound desc = could not find container \"865553cc212221dfc0a2512e3fa2e0fc000c515b403bcec127d87c052870b5a8\": container with ID starting with 865553cc212221dfc0a2512e3fa2e0fc000c515b403bcec127d87c052870b5a8 not found: ID does not exist" Jul 06 22:01:25.233114 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.233094 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj"] Jul 06 22:01:25.237252 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.237232 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-da1f6-predictor-564b4d98fc-8wvmj"] Jul 06 22:01:25.785975 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.785941 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5913d04b-be39-4226-8825-68074a5e74ea" path="/var/lib/kubelet/pods/5913d04b-be39-4226-8825-68074a5e74ea/volumes" Jul 06 22:01:25.786471 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:25.786451 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f20475b7-1ad4-43b0-b506-e72e28b106f4" path="/var/lib/kubelet/pods/f20475b7-1ad4-43b0-b506-e72e28b106f4/volumes" Jul 06 22:01:51.890638 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.890602 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d2pl7/must-gather-9clj5"] Jul 06 22:01:51.891025 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.890903 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f20475b7-1ad4-43b0-b506-e72e28b106f4" containerName="kube-rbac-proxy" Jul 06 22:01:51.891025 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.890915 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f20475b7-1ad4-43b0-b506-e72e28b106f4" containerName="kube-rbac-proxy" Jul 06 22:01:51.891025 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.890925 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f0d4bf0-2bec-413c-98d1-e3a5f264ab26" containerName="kserve-container" Jul 06 22:01:51.891025 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.890930 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f0d4bf0-2bec-413c-98d1-e3a5f264ab26" containerName="kserve-container" Jul 06 22:01:51.891025 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.890937 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="639574ec-8467-4bb3-9203-6168444f8331" containerName="kube-rbac-proxy" Jul 06 22:01:51.891025 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.890943 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="639574ec-8467-4bb3-9203-6168444f8331" containerName="kube-rbac-proxy" Jul 06 22:01:51.891025 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.890951 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5913d04b-be39-4226-8825-68074a5e74ea" containerName="kserve-container" Jul 06 22:01:51.891025 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.890956 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5913d04b-be39-4226-8825-68074a5e74ea" containerName="kserve-container" Jul 06 22:01:51.891025 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.890962 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5913d04b-be39-4226-8825-68074a5e74ea" containerName="kube-rbac-proxy" Jul 06 22:01:51.891025 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.890967 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5913d04b-be39-4226-8825-68074a5e74ea" containerName="kube-rbac-proxy" Jul 06 22:01:51.891025 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.890976 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="639574ec-8467-4bb3-9203-6168444f8331" containerName="kserve-container" Jul 06 22:01:51.891025 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.890981 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="639574ec-8467-4bb3-9203-6168444f8331" containerName="kserve-container" Jul 06 22:01:51.891025 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.890988 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e6066cd-42c1-4c15-b699-a0a438779f53" containerName="kube-rbac-proxy" Jul 06 22:01:51.891025 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.890993 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e6066cd-42c1-4c15-b699-a0a438779f53" containerName="kube-rbac-proxy" Jul 06 22:01:51.891025 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.891003 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f20475b7-1ad4-43b0-b506-e72e28b106f4" containerName="kserve-container" Jul 06 22:01:51.891025 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.891008 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f20475b7-1ad4-43b0-b506-e72e28b106f4" containerName="kserve-container" Jul 06 22:01:51.891025 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.891014 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c95e95-e959-46eb-b3b6-aa7f979d615f" containerName="kserve-container" Jul 06 22:01:51.891025 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.891019 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c95e95-e959-46eb-b3b6-aa7f979d615f" containerName="kserve-container" Jul 06 22:01:51.891025 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.891029 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f0d4bf0-2bec-413c-98d1-e3a5f264ab26" containerName="kube-rbac-proxy" Jul 06 22:01:51.891025 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.891035 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f0d4bf0-2bec-413c-98d1-e3a5f264ab26" containerName="kube-rbac-proxy" Jul 06 22:01:51.891815 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.891040 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e6066cd-42c1-4c15-b699-a0a438779f53" containerName="kserve-container" Jul 06 22:01:51.891815 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.891045 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e6066cd-42c1-4c15-b699-a0a438779f53" containerName="kserve-container" Jul 06 22:01:51.891815 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.891051 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c95e95-e959-46eb-b3b6-aa7f979d615f" containerName="kube-rbac-proxy" Jul 06 22:01:51.891815 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.891055 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c95e95-e959-46eb-b3b6-aa7f979d615f" containerName="kube-rbac-proxy" Jul 06 22:01:51.891815 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.891105 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e6066cd-42c1-4c15-b699-a0a438779f53" containerName="kserve-container" Jul 06 22:01:51.891815 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.891112 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f0d4bf0-2bec-413c-98d1-e3a5f264ab26" containerName="kube-rbac-proxy" Jul 06 22:01:51.891815 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.891121 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f20475b7-1ad4-43b0-b506-e72e28b106f4" containerName="kube-rbac-proxy" Jul 06 22:01:51.891815 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.891126 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e6066cd-42c1-4c15-b699-a0a438779f53" containerName="kube-rbac-proxy" Jul 06 22:01:51.891815 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.891131 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="76c95e95-e959-46eb-b3b6-aa7f979d615f" containerName="kube-rbac-proxy" Jul 06 22:01:51.891815 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.891135 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="639574ec-8467-4bb3-9203-6168444f8331" containerName="kube-rbac-proxy" Jul 06 22:01:51.891815 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.891140 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f20475b7-1ad4-43b0-b506-e72e28b106f4" containerName="kserve-container" Jul 06 22:01:51.891815 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.891146 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5913d04b-be39-4226-8825-68074a5e74ea" containerName="kserve-container" Jul 06 22:01:51.891815 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.891152 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="639574ec-8467-4bb3-9203-6168444f8331" containerName="kserve-container" Jul 06 22:01:51.891815 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.891157 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5913d04b-be39-4226-8825-68074a5e74ea" containerName="kube-rbac-proxy" Jul 06 22:01:51.891815 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.891163 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="76c95e95-e959-46eb-b3b6-aa7f979d615f" containerName="kserve-container" Jul 06 22:01:51.891815 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.891168 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f0d4bf0-2bec-413c-98d1-e3a5f264ab26" containerName="kserve-container" Jul 06 22:01:51.893780 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.893762 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d2pl7/must-gather-9clj5" Jul 06 22:01:51.896003 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.895982 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d2pl7\"/\"openshift-service-ca.crt\"" Jul 06 22:01:51.896113 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.896013 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d2pl7\"/\"kube-root-ca.crt\"" Jul 06 22:01:51.897035 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.897020 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-d2pl7\"/\"default-dockercfg-hwbkv\"" Jul 06 22:01:51.904032 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.904012 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d2pl7/must-gather-9clj5"] Jul 06 22:01:51.946477 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.946452 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6jdf\" (UniqueName: \"kubernetes.io/projected/a0643bbd-e845-46cc-984a-382fa56bc43b-kube-api-access-f6jdf\") pod \"must-gather-9clj5\" (UID: \"a0643bbd-e845-46cc-984a-382fa56bc43b\") " pod="openshift-must-gather-d2pl7/must-gather-9clj5" Jul 06 22:01:51.946594 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:51.946510 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a0643bbd-e845-46cc-984a-382fa56bc43b-must-gather-output\") pod \"must-gather-9clj5\" (UID: \"a0643bbd-e845-46cc-984a-382fa56bc43b\") " pod="openshift-must-gather-d2pl7/must-gather-9clj5" Jul 06 22:01:52.047719 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:52.047688 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6jdf\" (UniqueName: \"kubernetes.io/projected/a0643bbd-e845-46cc-984a-382fa56bc43b-kube-api-access-f6jdf\") pod \"must-gather-9clj5\" (UID: \"a0643bbd-e845-46cc-984a-382fa56bc43b\") " pod="openshift-must-gather-d2pl7/must-gather-9clj5" Jul 06 22:01:52.047850 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:52.047755 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a0643bbd-e845-46cc-984a-382fa56bc43b-must-gather-output\") pod \"must-gather-9clj5\" (UID: \"a0643bbd-e845-46cc-984a-382fa56bc43b\") " pod="openshift-must-gather-d2pl7/must-gather-9clj5" Jul 06 22:01:52.048054 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:52.048034 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a0643bbd-e845-46cc-984a-382fa56bc43b-must-gather-output\") pod \"must-gather-9clj5\" (UID: \"a0643bbd-e845-46cc-984a-382fa56bc43b\") " pod="openshift-must-gather-d2pl7/must-gather-9clj5" Jul 06 22:01:52.055264 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:52.055244 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6jdf\" (UniqueName: \"kubernetes.io/projected/a0643bbd-e845-46cc-984a-382fa56bc43b-kube-api-access-f6jdf\") pod \"must-gather-9clj5\" (UID: \"a0643bbd-e845-46cc-984a-382fa56bc43b\") " pod="openshift-must-gather-d2pl7/must-gather-9clj5" Jul 06 22:01:52.202462 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:52.202376 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d2pl7/must-gather-9clj5" Jul 06 22:01:52.326836 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:52.326806 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d2pl7/must-gather-9clj5"] Jul 06 22:01:52.330032 ip-10-0-140-54 kubenswrapper[2581]: W0706 22:01:52.330003 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0643bbd_e845_46cc_984a_382fa56bc43b.slice/crio-8c1c24077ca6b244c0957efc18ea4c68467183590bd1a1a51c41b65ce0ab006a WatchSource:0}: Error finding container 8c1c24077ca6b244c0957efc18ea4c68467183590bd1a1a51c41b65ce0ab006a: Status 404 returned error can't find the container with id 8c1c24077ca6b244c0957efc18ea4c68467183590bd1a1a51c41b65ce0ab006a Jul 06 22:01:52.331858 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:52.331839 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:01:53.309720 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:53.309687 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d2pl7/must-gather-9clj5" event={"ID":"a0643bbd-e845-46cc-984a-382fa56bc43b","Type":"ContainerStarted","Data":"26ae7b833858bac0a67ff61cc8dc766591d57d39cc01a17544d344aa2f4b4786"} Jul 06 22:01:53.310052 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:53.309739 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d2pl7/must-gather-9clj5" event={"ID":"a0643bbd-e845-46cc-984a-382fa56bc43b","Type":"ContainerStarted","Data":"8c1c24077ca6b244c0957efc18ea4c68467183590bd1a1a51c41b65ce0ab006a"} Jul 06 22:01:54.317688 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:54.317649 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d2pl7/must-gather-9clj5" event={"ID":"a0643bbd-e845-46cc-984a-382fa56bc43b","Type":"ContainerStarted","Data":"c723b3d3e708059c8b5f2d321ee51ea324a1cf31a21ada2cdf3be392d419aac6"} Jul 06 22:01:54.334676 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:54.334626 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-d2pl7/must-gather-9clj5" podStartSLOduration=2.56410258 podStartE2EDuration="3.33461129s" totalImagesPullingTime="770.50871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:01:54.332998779 +0000 UTC m=+2939.095911784" watchObservedRunningTime="2026-07-06 22:01:54.33461129 +0000 UTC m=+2939.097524293" Jul 06 22:01:54.524393 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:54.524361 2581 ???:1] "http: TLS handshake error from 10.0.136.12:56882: EOF" Jul 06 22:01:54.533813 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:54.533783 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zn2lg_e4aacd25-63b3-4414-8e53-5d418169367b/global-pull-secret-syncer/0.log" Jul 06 22:01:54.621149 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:54.621112 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mxfzl_d38a1184-76ff-4dc4-bacf-bbf90f547ca4/konnectivity-agent/0.log" Jul 06 22:01:54.711534 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:54.711505 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-54.ec2.internal_b78cd9a8597733097f5408183b6fd42d/haproxy/0.log" Jul 06 22:01:59.200242 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:59.200211 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a3a65cf2-ed8f-4b79-b37d-0f669b835b7d/alertmanager/0.log" Jul 06 22:01:59.229668 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:59.229595 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a3a65cf2-ed8f-4b79-b37d-0f669b835b7d/config-reloader/0.log" Jul 06 22:01:59.262435 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:59.262290 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a3a65cf2-ed8f-4b79-b37d-0f669b835b7d/kube-rbac-proxy-web/0.log" Jul 06 22:01:59.289459 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:59.289434 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a3a65cf2-ed8f-4b79-b37d-0f669b835b7d/kube-rbac-proxy/0.log" Jul 06 22:01:59.315762 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:59.315732 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a3a65cf2-ed8f-4b79-b37d-0f669b835b7d/kube-rbac-proxy-metric/0.log" Jul 06 22:01:59.336483 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:59.336458 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a3a65cf2-ed8f-4b79-b37d-0f669b835b7d/prom-label-proxy/0.log" Jul 06 22:01:59.359856 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:59.359781 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a3a65cf2-ed8f-4b79-b37d-0f669b835b7d/init-config-reloader/0.log" Jul 06 22:01:59.740621 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:59.740579 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ssmlt_2b7070b6-7594-44c9-9d25-6ef3366255bf/node-exporter/0.log" Jul 06 22:01:59.763687 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:59.763646 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ssmlt_2b7070b6-7594-44c9-9d25-6ef3366255bf/kube-rbac-proxy/0.log" Jul 06 22:01:59.783196 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:01:59.783170 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ssmlt_2b7070b6-7594-44c9-9d25-6ef3366255bf/init-textfile/0.log" Jul 06 22:02:00.168468 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:00.168437 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f4d7fcb8b-7pjxh_43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1/telemeter-client/0.log" Jul 06 22:02:00.197625 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:00.197580 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f4d7fcb8b-7pjxh_43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1/reload/0.log" Jul 06 22:02:00.228659 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:00.228632 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f4d7fcb8b-7pjxh_43d4bdd3-6182-4ada-90d6-4bdfcfff4eb1/kube-rbac-proxy/0.log" Jul 06 22:02:00.272563 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:00.272297 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-886dfc74b-q78n8_b29fc180-58a8-477f-928b-3659cc343965/thanos-query/0.log" Jul 06 22:02:00.298205 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:00.298173 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-886dfc74b-q78n8_b29fc180-58a8-477f-928b-3659cc343965/kube-rbac-proxy-web/0.log" Jul 06 22:02:00.323566 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:00.323541 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-886dfc74b-q78n8_b29fc180-58a8-477f-928b-3659cc343965/kube-rbac-proxy/0.log" Jul 06 22:02:00.345826 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:00.345795 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-886dfc74b-q78n8_b29fc180-58a8-477f-928b-3659cc343965/prom-label-proxy/0.log" Jul 06 22:02:00.367482 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:00.367399 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-886dfc74b-q78n8_b29fc180-58a8-477f-928b-3659cc343965/kube-rbac-proxy-rules/0.log" Jul 06 22:02:00.391896 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:00.391864 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-886dfc74b-q78n8_b29fc180-58a8-477f-928b-3659cc343965/kube-rbac-proxy-metrics/0.log" Jul 06 22:02:02.445981 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:02.445946 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-794468d945-9tpj5_0c22b3e4-48ec-4632-b8d3-d8725f9a7c08/console/0.log" Jul 06 22:02:02.826270 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:02.826164 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n"] Jul 06 22:02:02.832027 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:02.831989 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n" Jul 06 22:02:02.842883 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:02.842846 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n"] Jul 06 22:02:02.943894 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:02.943844 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35edce3c-815e-4408-a870-dc7e3db9cc99-sys\") pod \"perf-node-gather-daemonset-bq95n\" (UID: \"35edce3c-815e-4408-a870-dc7e3db9cc99\") " pod="openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n" Jul 06 22:02:02.944070 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:02.943949 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/35edce3c-815e-4408-a870-dc7e3db9cc99-podres\") pod \"perf-node-gather-daemonset-bq95n\" (UID: \"35edce3c-815e-4408-a870-dc7e3db9cc99\") " pod="openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n" Jul 06 22:02:02.944152 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:02.944081 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/35edce3c-815e-4408-a870-dc7e3db9cc99-proc\") pod \"perf-node-gather-daemonset-bq95n\" (UID: \"35edce3c-815e-4408-a870-dc7e3db9cc99\") " pod="openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n" Jul 06 22:02:02.944152 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:02.944123 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sztbs\" (UniqueName: \"kubernetes.io/projected/35edce3c-815e-4408-a870-dc7e3db9cc99-kube-api-access-sztbs\") pod \"perf-node-gather-daemonset-bq95n\" (UID: \"35edce3c-815e-4408-a870-dc7e3db9cc99\") " pod="openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n" Jul 06 22:02:02.944272 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:02.944225 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35edce3c-815e-4408-a870-dc7e3db9cc99-lib-modules\") pod \"perf-node-gather-daemonset-bq95n\" (UID: \"35edce3c-815e-4408-a870-dc7e3db9cc99\") " pod="openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n" Jul 06 22:02:03.032078 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:03.032046 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-629h4_68f8fa10-1463-4367-be99-23850541f8fc/volume-data-source-validator/0.log" Jul 06 22:02:03.045057 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:03.045024 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/35edce3c-815e-4408-a870-dc7e3db9cc99-proc\") pod \"perf-node-gather-daemonset-bq95n\" (UID: \"35edce3c-815e-4408-a870-dc7e3db9cc99\") " pod="openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n" Jul 06 22:02:03.045217 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:03.045067 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sztbs\" (UniqueName: \"kubernetes.io/projected/35edce3c-815e-4408-a870-dc7e3db9cc99-kube-api-access-sztbs\") pod \"perf-node-gather-daemonset-bq95n\" (UID: \"35edce3c-815e-4408-a870-dc7e3db9cc99\") " pod="openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n" Jul 06 22:02:03.045217 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:03.045104 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35edce3c-815e-4408-a870-dc7e3db9cc99-lib-modules\") pod \"perf-node-gather-daemonset-bq95n\" (UID: \"35edce3c-815e-4408-a870-dc7e3db9cc99\") " pod="openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n" Jul 06 22:02:03.045217 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:03.045129 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35edce3c-815e-4408-a870-dc7e3db9cc99-sys\") pod \"perf-node-gather-daemonset-bq95n\" (UID: \"35edce3c-815e-4408-a870-dc7e3db9cc99\") " pod="openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n" Jul 06 22:02:03.045217 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:03.045139 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/35edce3c-815e-4408-a870-dc7e3db9cc99-proc\") pod \"perf-node-gather-daemonset-bq95n\" (UID: \"35edce3c-815e-4408-a870-dc7e3db9cc99\") " pod="openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n" Jul 06 22:02:03.045217 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:03.045153 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/35edce3c-815e-4408-a870-dc7e3db9cc99-podres\") pod \"perf-node-gather-daemonset-bq95n\" (UID: \"35edce3c-815e-4408-a870-dc7e3db9cc99\") " pod="openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n" Jul 06 22:02:03.045409 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:03.045273 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35edce3c-815e-4408-a870-dc7e3db9cc99-sys\") pod \"perf-node-gather-daemonset-bq95n\" (UID: \"35edce3c-815e-4408-a870-dc7e3db9cc99\") " pod="openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n" Jul 06 22:02:03.045409 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:03.045294 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/35edce3c-815e-4408-a870-dc7e3db9cc99-podres\") pod \"perf-node-gather-daemonset-bq95n\" (UID: \"35edce3c-815e-4408-a870-dc7e3db9cc99\") " pod="openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n" Jul 06 22:02:03.045409 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:03.045381 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35edce3c-815e-4408-a870-dc7e3db9cc99-lib-modules\") pod \"perf-node-gather-daemonset-bq95n\" (UID: \"35edce3c-815e-4408-a870-dc7e3db9cc99\") " pod="openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n" Jul 06 22:02:03.056052 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:03.056021 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sztbs\" (UniqueName: \"kubernetes.io/projected/35edce3c-815e-4408-a870-dc7e3db9cc99-kube-api-access-sztbs\") pod \"perf-node-gather-daemonset-bq95n\" (UID: \"35edce3c-815e-4408-a870-dc7e3db9cc99\") " pod="openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n" Jul 06 22:02:03.144577 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:03.144509 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n" Jul 06 22:02:03.299144 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:03.299117 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n"] Jul 06 22:02:03.301832 ip-10-0-140-54 kubenswrapper[2581]: W0706 22:02:03.301767 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod35edce3c_815e_4408_a870_dc7e3db9cc99.slice/crio-957a2b47fcaf78f59dee95d2c2333258bd6780d5d41845bfa54bbcbb2b1fb71b WatchSource:0}: Error finding container 957a2b47fcaf78f59dee95d2c2333258bd6780d5d41845bfa54bbcbb2b1fb71b: Status 404 returned error can't find the container with id 957a2b47fcaf78f59dee95d2c2333258bd6780d5d41845bfa54bbcbb2b1fb71b Jul 06 22:02:03.367227 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:03.367190 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n" event={"ID":"35edce3c-815e-4408-a870-dc7e3db9cc99","Type":"ContainerStarted","Data":"957a2b47fcaf78f59dee95d2c2333258bd6780d5d41845bfa54bbcbb2b1fb71b"} Jul 06 22:02:03.966069 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:03.966026 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sstxl_c160ad01-8d36-4567-945d-fd6810e76537/dns/0.log" Jul 06 22:02:03.986490 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:03.986457 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sstxl_c160ad01-8d36-4567-945d-fd6810e76537/kube-rbac-proxy/0.log" Jul 06 22:02:04.068293 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:04.068261 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2q95f_febe62f7-bf85-42c1-8902-c4dabb9ba553/dns-node-resolver/0.log" Jul 06 22:02:04.372761 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:04.372722 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n" event={"ID":"35edce3c-815e-4408-a870-dc7e3db9cc99","Type":"ContainerStarted","Data":"76ac8cf6e1ca18209c941fa6607e5cb1613f0c63aaff8149f8b9a0ac4ee4e797"} Jul 06 22:02:04.372971 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:04.372851 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n" Jul 06 22:02:04.654275 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:04.654200 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lds24_08f54334-35ee-460d-ae74-75618e3e13c5/node-ca/0.log" Jul 06 22:02:05.462922 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:05.462890 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-548d5d67c5-5qdmw_ad697c3c-f1a2-4209-827e-76d6164bd893/router/0.log" Jul 06 22:02:05.807837 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:05.807757 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jxz5h_704cb9c1-1b35-419a-90f0-0cf0dd5498f5/serve-healthcheck-canary/0.log" Jul 06 22:02:06.170591 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:06.170561 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-cnsh7_2a156ee3-56b6-45e1-80de-debe10e30cca/insights-operator/0.log" Jul 06 22:02:06.171649 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:06.171620 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-cnsh7_2a156ee3-56b6-45e1-80de-debe10e30cca/insights-operator/1.log" Jul 06 22:02:06.193020 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:06.192995 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b9wqf_4c41ac7d-dd13-4ea9-beec-7dfa17111ba1/kube-rbac-proxy/0.log" Jul 06 22:02:06.215802 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:06.215773 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b9wqf_4c41ac7d-dd13-4ea9-beec-7dfa17111ba1/exporter/0.log" Jul 06 22:02:06.235567 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:06.235544 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b9wqf_4c41ac7d-dd13-4ea9-beec-7dfa17111ba1/extractor/0.log" Jul 06 22:02:08.570404 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:08.570372 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-dvpz8_5358db15-8985-486f-9106-5e3e1eac81e7/manager/0.log" Jul 06 22:02:08.664494 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:08.664467 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-xfxpm_4a1aa2ef-a835-40cf-81b6-24c6f134fc52/seaweedfs/0.log" Jul 06 22:02:10.389492 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:10.389154 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n" Jul 06 22:02:10.407735 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:10.407684 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-d2pl7/perf-node-gather-daemonset-bq95n" podStartSLOduration=8.407665151 podStartE2EDuration="8.407665151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:02:04.390110493 +0000 UTC m=+2949.153023500" watchObservedRunningTime="2026-07-06 22:02:10.407665151 +0000 UTC m=+2955.170578158" Jul 06 22:02:12.958405 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:12.958290 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-2tsvl_b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be/kube-storage-version-migrator-operator/1.log" Jul 06 22:02:12.960055 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:12.960024 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-2tsvl_b8b6dfda-6ca5-4f85-a0f4-ef6a4e1b60be/kube-storage-version-migrator-operator/0.log" Jul 06 22:02:14.149562 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:14.149531 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x4grf_b03516a3-66d9-4cd4-b814-dc23dd3bd9a1/kube-multus-additional-cni-plugins/0.log" Jul 06 22:02:14.168844 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:14.168817 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x4grf_b03516a3-66d9-4cd4-b814-dc23dd3bd9a1/egress-router-binary-copy/0.log" Jul 06 22:02:14.187802 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:14.187775 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x4grf_b03516a3-66d9-4cd4-b814-dc23dd3bd9a1/cni-plugins/0.log" Jul 06 22:02:14.209671 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:14.209642 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x4grf_b03516a3-66d9-4cd4-b814-dc23dd3bd9a1/bond-cni-plugin/0.log" Jul 06 22:02:14.227812 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:14.227786 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x4grf_b03516a3-66d9-4cd4-b814-dc23dd3bd9a1/routeoverride-cni/0.log" Jul 06 22:02:14.248252 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:14.248170 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x4grf_b03516a3-66d9-4cd4-b814-dc23dd3bd9a1/whereabouts-cni-bincopy/0.log" Jul 06 22:02:14.267447 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:14.267419 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x4grf_b03516a3-66d9-4cd4-b814-dc23dd3bd9a1/whereabouts-cni/0.log" Jul 06 22:02:14.360701 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:14.360675 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xq8wc_0eafbe5d-cd16-448b-b3c0-156ca0c03981/kube-multus/0.log" Jul 06 22:02:14.380267 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:14.380240 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-79rs9_e4f4e605-6394-4afd-8ef5-55c6fe78cb07/network-metrics-daemon/0.log" Jul 06 22:02:14.396298 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:14.396279 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-79rs9_e4f4e605-6394-4afd-8ef5-55c6fe78cb07/kube-rbac-proxy/0.log" Jul 06 22:02:15.164710 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:15.164668 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq68j_a276f91f-2f3c-4f2c-85ff-f53d4bc6c713/ovn-controller/0.log" Jul 06 22:02:15.184243 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:15.184173 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq68j_a276f91f-2f3c-4f2c-85ff-f53d4bc6c713/ovn-acl-logging/0.log" Jul 06 22:02:15.207726 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:15.207680 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq68j_a276f91f-2f3c-4f2c-85ff-f53d4bc6c713/kube-rbac-proxy-node/0.log" Jul 06 22:02:15.226605 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:15.226574 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq68j_a276f91f-2f3c-4f2c-85ff-f53d4bc6c713/kube-rbac-proxy-ovn-metrics/0.log" Jul 06 22:02:15.242318 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:15.242290 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq68j_a276f91f-2f3c-4f2c-85ff-f53d4bc6c713/northd/0.log" Jul 06 22:02:15.263068 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:15.263041 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq68j_a276f91f-2f3c-4f2c-85ff-f53d4bc6c713/nbdb/0.log" Jul 06 22:02:15.282781 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:15.282756 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq68j_a276f91f-2f3c-4f2c-85ff-f53d4bc6c713/sbdb/0.log" Jul 06 22:02:15.351075 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:15.351034 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq68j_a276f91f-2f3c-4f2c-85ff-f53d4bc6c713/ovnkube-controller/0.log" Jul 06 22:02:16.709430 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:16.709402 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-989f8b8dc-2m2h6_a0b9e0da-5047-4951-8240-59dfad0b3efa/check-endpoints/0.log" Jul 06 22:02:16.781190 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:16.781155 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-wjc92_b4d5a450-11db-44a5-bb22-35f668de29c7/network-check-target-container/0.log" Jul 06 22:02:17.649459 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:17.649427 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-nqzqb_75b9c669-e2a1-4c31-b58c-905b30293f0a/iptables-alerter/0.log" Jul 06 22:02:18.298375 ip-10-0-140-54 kubenswrapper[2581]: I0706 22:02:18.298328 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-xngg9_e27e6563-2401-444f-962e-575b5e9cb93f/tuned/0.log"