May 25 13:04:43.860559 ip-10-0-139-18 systemd[1]: Starting Kubernetes Kubelet...
May 25 13:04:44.405130 ip-10-0-139-18 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 13:04:44.405130 ip-10-0-139-18 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 25 13:04:44.405130 ip-10-0-139-18 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 13:04:44.405130 ip-10-0-139-18 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 25 13:04:44.405130 ip-10-0-139-18 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 13:04:44.405130 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.368832    2569 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 25 13:04:44.405130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379540    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:04:44.405130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379558    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:04:44.405130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379562    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:04:44.405130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379565    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:04:44.405130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379569    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:04:44.405130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379572    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:04:44.405130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379575    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:04:44.405130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379581    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:04:44.406473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379585    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:04:44.406473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379588    2569 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:04:44.406473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379591    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:04:44.406473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379595    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:04:44.406473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379599    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:04:44.406473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379601    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:04:44.406473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379605    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:04:44.406473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379608    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:04:44.406473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379611    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:04:44.406473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379614    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:04:44.406473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379617    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:04:44.406473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379620    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:04:44.406473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379623    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:04:44.406473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379626    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:04:44.406473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379629    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:04:44.406473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379631    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:04:44.406473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379634    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:04:44.406473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379637    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:04:44.406473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379640    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:04:44.406473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379642    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:04:44.407030 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379645    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:04:44.407030 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379647    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:04:44.407030 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379650    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:04:44.407030 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379653    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:04:44.407030 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379656    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:04:44.407030 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379659    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:04:44.407030 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379661    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:04:44.407030 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379664    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:04:44.407030 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379667    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:04:44.407030 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379669    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:04:44.407030 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379672    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:04:44.407030 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379674    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:04:44.407030 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379677    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:04:44.407030 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379680    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:04:44.407030 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379682    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:04:44.407030 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379685    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:04:44.407030 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379687    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:04:44.407030 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379690    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:04:44.407030 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379692    2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:04:44.407030 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379695    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:04:44.407807 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379698    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:04:44.407807 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379701    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:04:44.407807 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379704    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:04:44.407807 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379707    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:04:44.407807 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379710    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:04:44.407807 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379712    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:04:44.407807 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379715    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:04:44.407807 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379717    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:04:44.407807 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379720    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:04:44.407807 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379723    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:04:44.407807 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379725    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:04:44.407807 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379728    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:04:44.407807 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379731    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:04:44.407807 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379733    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:04:44.407807 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379736    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:04:44.407807 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379738    2569 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:04:44.407807 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379741    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:04:44.407807 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379743    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:04:44.407807 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379746    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:04:44.407807 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379748    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:04:44.408377 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379751    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:04:44.408377 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379753    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:04:44.408377 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379756    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:04:44.408377 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379758    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:04:44.408377 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379761    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:04:44.408377 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379764    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:04:44.408377 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379766    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:04:44.408377 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379769    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:04:44.408377 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379771    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:04:44.408377 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379774    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:04:44.408377 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379786    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:04:44.408377 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379790    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:04:44.408377 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379793    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:04:44.408377 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379795    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:04:44.408377 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379798    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:04:44.408377 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379802    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:04:44.408377 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379804    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:04:44.408377 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.379807    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:04:44.408377 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382475    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:04:44.408377 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382484    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:04:44.409131 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382487    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:04:44.409131 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382490    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:04:44.409131 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382493    2569 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:04:44.409131 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382496    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:04:44.409131 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382499    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:04:44.409131 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382502    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:04:44.409131 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382505    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:04:44.409131 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382507    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:04:44.409131 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382510    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:04:44.409131 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382513    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:04:44.409131 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382516    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:04:44.409131 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382519    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:04:44.409131 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382521    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:04:44.409131 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382524    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:04:44.409131 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382527    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:04:44.409131 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382529    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:04:44.409131 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382532    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:04:44.409131 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382534    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:04:44.409131 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382537    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:04:44.409131 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382539    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:04:44.409709 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382542    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:04:44.409709 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382545    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:04:44.409709 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382549    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:04:44.409709 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382552    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:04:44.409709 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382555    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:04:44.409709 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382558    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:04:44.409709 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382561    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:04:44.409709 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382563    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:04:44.409709 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382566    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:04:44.409709 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382569    2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:04:44.409709 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382571    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:04:44.409709 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382574    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:04:44.409709 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382577    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:04:44.409709 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382579    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:04:44.409709 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382583    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:04:44.409709 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382585    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:04:44.409709 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382588    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:04:44.409709 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382591    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:04:44.409709 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382593    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:04:44.409709 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382597    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:04:44.410485 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382599    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:04:44.410485 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382602    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:04:44.410485 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382605    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:04:44.410485 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382608    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:04:44.410485 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382611    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:04:44.410485 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382615    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:04:44.410485 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382619    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:04:44.410485 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382622    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:04:44.410485 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382625    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:04:44.410485 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382627    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:04:44.410485 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382630    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:04:44.410485 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382632    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:04:44.410485 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382635    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:04:44.410485 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382637    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:04:44.410485 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382640    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:04:44.410485 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382644    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:04:44.410485 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382647    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:04:44.410485 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382650    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:04:44.410485 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382652    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:04:44.411028 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382655    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:04:44.411028 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382658    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:04:44.411028 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382661    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:04:44.411028 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382663    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:04:44.411028 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382666    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:04:44.411028 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382668    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:04:44.411028 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382671    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:04:44.411028 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382674    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:04:44.411028 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382676    2569 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:04:44.411028 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382679    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:04:44.411028 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382682    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:04:44.411028 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382686    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:04:44.411028 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382690    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:04:44.411028 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382693    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:04:44.411028 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382696    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:04:44.411028 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382700    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:04:44.411028 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382703    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:04:44.411028 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382706    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:04:44.411028 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382708    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:04:44.411028 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382711    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382714    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382716    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382719    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382721    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.382724    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382794    2569 flags.go:64] FLAG: --address="0.0.0.0"
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382802    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382809    2569 flags.go:64] FLAG: --anonymous-auth="true"
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382813    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382819    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382822    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382826    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382831    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382836    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382840    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382843    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382847    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382850    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382853    2569 flags.go:64] FLAG: --cgroup-root=""
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382856    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382859    2569 flags.go:64] FLAG: --client-ca-file=""
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382861    2569 flags.go:64] FLAG: --cloud-config=""
May 25 13:04:44.411727 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382864    2569 flags.go:64] FLAG: --cloud-provider="external"
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382867    2569 flags.go:64] FLAG: --cluster-dns="[]"
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382871    2569 flags.go:64] FLAG: --cluster-domain=""
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382874    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382877    2569 flags.go:64] FLAG: --config-dir=""
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382881    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382884    2569 flags.go:64] FLAG: --container-log-max-files="5"
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382888    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382891    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382895    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382898    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382901    2569 flags.go:64] FLAG: --contention-profiling="false"
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382904    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382908    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382915    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382918    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382922    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382926    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382929    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382932    2569 flags.go:64] FLAG: --enable-load-reader="false"
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382936    2569 flags.go:64] FLAG: --enable-server="true"
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382939    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382944    2569 flags.go:64] FLAG: --event-burst="100"
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382948    2569 flags.go:64] FLAG: --event-qps="50"
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382951    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 25 13:04:44.412379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382955    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382958    2569 flags.go:64] FLAG: --eviction-hard=""
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382962    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382965    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382968    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382971    2569 flags.go:64] FLAG: --eviction-soft=""
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382974    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382977    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382979    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382983    2569 flags.go:64] FLAG: --experimental-mounter-path=""
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382986    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382989    2569 flags.go:64] FLAG: --fail-swap-on="true"
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382992    2569 flags.go:64] FLAG: --feature-gates=""
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382996    2569 flags.go:64] FLAG: --file-check-frequency="20s"
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.382999    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383002    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383005    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383008    2569 flags.go:64] FLAG: --healthz-port="10248"
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383011    2569 flags.go:64] FLAG: --help="false"
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383014    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-139-18.ec2.internal"
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383017    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383020    2569 flags.go:64] FLAG: --http-check-frequency="20s"
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383023    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383026    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 25 13:04:44.562530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383030    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 25 13:04:44.491333 ip-10-0-139-18 systemd[1]: Started Kubernetes Kubelet.
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383033    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383036    2569 flags.go:64] FLAG: --image-service-endpoint=""
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383039    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383042    2569 flags.go:64] FLAG: --kube-api-burst="100"
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383045    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383054    2569 flags.go:64] FLAG: --kube-api-qps="50"
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383057    2569 flags.go:64] FLAG: --kube-reserved=""
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383061    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383063    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383067    2569 flags.go:64] FLAG: --kubelet-cgroups=""
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383070    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383072    2569 flags.go:64] FLAG: --lock-file=""
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383075    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383079    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383082    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383087    2569 flags.go:64] FLAG: --log-json-split-stream="false"
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383090    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383093    2569 flags.go:64] FLAG: --log-text-split-stream="false"
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383096    2569 flags.go:64] FLAG: --logging-format="text"
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383099    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383102    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383106    2569 flags.go:64] FLAG: --manifest-url=""
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383108    2569 flags.go:64] FLAG: --manifest-url-header=""
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383113    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383116    2569 flags.go:64] FLAG: --max-open-files="1000000"
May 25 13:04:44.563663 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383120    2569 flags.go:64] FLAG: --max-pods="110"
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383123    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383126    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383129    2569 flags.go:64] FLAG: --memory-manager-policy="None"
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383132    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383135    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383138    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383141    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383149    2569 flags.go:64] FLAG: --node-status-max-images="50"
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383152    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383155    2569 flags.go:64] FLAG: --oom-score-adj="-999"
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383159    2569 flags.go:64] FLAG: --pod-cidr=""
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383163    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383189    2569 flags.go:64] FLAG: --pod-manifest-path=""
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383192    2569 flags.go:64] FLAG: --pod-max-pids="-1"
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383195    2569 flags.go:64] FLAG: --pods-per-core="0"
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383198    2569 flags.go:64] FLAG: --port="10250"
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383201    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383204    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0172ea15eccf76324"
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383207    2569 flags.go:64] FLAG: --qos-reserved=""
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383210    2569 flags.go:64] FLAG: --read-only-port="10255"
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383214    2569 flags.go:64] FLAG: --register-node="true"
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383217    2569 flags.go:64] FLAG: --register-schedulable="true"
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383220    2569 flags.go:64] FLAG: --register-with-taints=""
May 25 13:04:44.565077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383223    2569 flags.go:64] FLAG: --registry-burst="10"
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383226    2569 flags.go:64] FLAG: --registry-qps="5"
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383229    2569 flags.go:64] FLAG: --reserved-cpus=""
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383232    2569 flags.go:64] FLAG: --reserved-memory=""
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383238    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383242    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383245    2569 flags.go:64] FLAG: --rotate-certificates="false"
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383248    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383251    2569 flags.go:64] FLAG: --runonce="false"
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383253    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383257    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383260    2569 flags.go:64] FLAG: --seccomp-default="false"
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383263    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383265    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383269    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383272    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383275    2569 flags.go:64] FLAG: --storage-driver-password="root"
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383278    2569 flags.go:64] FLAG: --storage-driver-secure="false"
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383281    2569 flags.go:64] FLAG: --storage-driver-table="stats"
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383284    2569 flags.go:64] FLAG: --storage-driver-user="root"
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383289    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383293    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383296    2569 flags.go:64] FLAG: --system-cgroups=""
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383299    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383304    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
May 25 13:04:44.566319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383307    2569 flags.go:64] FLAG: --tls-cert-file=""
May 25 13:04:44.567577 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383310    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 25 13:04:44.567577 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383314    2569 flags.go:64] FLAG: --tls-min-version=""
May 25 13:04:44.567577 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383317    2569 flags.go:64] FLAG: --tls-private-key-file=""
May 25 13:04:44.567577 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383320    2569 flags.go:64] FLAG: --topology-manager-policy="none"
May 25 13:04:44.567577 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383323    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
May 25 13:04:44.567577 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383325    2569 flags.go:64] FLAG: --topology-manager-scope="container"
May 25 13:04:44.567577 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383328    2569 flags.go:64] FLAG: --v="2"
May 25 13:04:44.567577 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383333    2569 flags.go:64] FLAG: --version="false"
May 25 13:04:44.567577 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383337    2569 flags.go:64] FLAG: --vmodule=""
May 25 13:04:44.567577 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383341    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 25 13:04:44.567577 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.383344    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 25 13:04:44.567577 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383436    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:04:44.567577 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383439    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:04:44.567577 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383451    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:04:44.567577 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383455    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:04:44.567577 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383458    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:04:44.567577 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383461    2569 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:04:44.567577 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383464    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:04:44.567577 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383467    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:04:44.567577 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383471    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:04:44.567577 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383473    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:04:44.567577 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383476    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:04:44.570130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383479    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:04:44.570130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383482    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:04:44.570130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383485    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:04:44.570130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383487    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:04:44.570130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383495    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:04:44.570130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383499    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:04:44.570130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383502    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:04:44.570130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383504    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:04:44.570130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383507    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:04:44.570130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383510    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:04:44.570130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383512    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:04:44.570130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383515    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:04:44.570130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383518    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:04:44.570130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383520    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:04:44.570130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383523    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:04:44.570130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383526    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:04:44.570130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383530    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:04:44.570130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383533    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:04:44.570130 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383536    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:04:44.606272 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383539    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:04:44.606272 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383542    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:04:44.606272 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383544    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:04:44.606272 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383547    2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:04:44.606272 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383550    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:04:44.606272 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383552    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:04:44.606272 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383555    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:04:44.606272 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383558    2569 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:04:44.606272 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383560    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:04:44.606272 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383563    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:04:44.606272 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383566    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:04:44.606272 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383569    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:04:44.606272 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383571    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:04:44.606272 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383574    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:04:44.606272 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383577    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:04:44.606272 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383579    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:04:44.606272 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383582    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:04:44.606272 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383587    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:04:44.606272 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383590    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:04:44.606272 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383593    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:04:44.607005 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383596    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:04:44.607005 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383598    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:04:44.607005 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383601    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:04:44.607005 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383604    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:04:44.607005 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383606    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:04:44.607005 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383609    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:04:44.607005 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383611    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:04:44.607005 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383614    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:04:44.607005 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383616    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:04:44.607005 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383619    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:04:44.607005 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383622    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:04:44.607005 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383624    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:04:44.607005 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383627    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:04:44.607005 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383629    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:04:44.607005 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383632    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:04:44.607005 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383635    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:04:44.607005 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383637    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:04:44.607005 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383640    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:04:44.607005 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383642    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:04:44.607005 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383645    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:04:44.607005 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383648    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:04:44.607875 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383650    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:04:44.607875 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383653    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:04:44.607875 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383655    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:04:44.607875 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383658    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:04:44.607875 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383660    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:04:44.607875 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383663    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:04:44.607875 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383665    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:04:44.607875 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383668    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:04:44.607875 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383673    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:04:44.607875 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383676    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:04:44.607875 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383678    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:04:44.607875 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383681    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:04:44.607875 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383684    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:04:44.607875 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383687    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:04:44.607875 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.383689    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:04:44.607875 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.384626    2569 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 13:04:44.608473 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.392056    2569 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 25 13:04:44.608473 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.392075    2569 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 25 13:04:44.608473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392133    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:04:44.608473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392139    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:04:44.608473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392144    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:04:44.608473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392149    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:04:44.608473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392154    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:04:44.608473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392159    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:04:44.608473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392161    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:04:44.608473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392177    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:04:44.608473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392180    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:04:44.608473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392183    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:04:44.608473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392186    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:04:44.608473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392189    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:04:44.608473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392191    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:04:44.608473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392194    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:04:44.608473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392196    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:04:44.608473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392199    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:04:44.608473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392202    2569 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:04:44.608473 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392204    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:04:44.609341 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392207    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:04:44.609341 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392210    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:04:44.609341 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392213    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:04:44.609341 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392215    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:04:44.609341 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392218    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:04:44.609341 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392222    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:04:44.609341 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392226    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:04:44.609341 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392230    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:04:44.609341 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392236    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:04:44.609341 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392240    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:04:44.609341 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392243    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:04:44.609341 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392246    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:04:44.609341 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392249    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:04:44.609341 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392253    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:04:44.609341 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392255    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:04:44.609341 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392258    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:04:44.609341 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392261    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:04:44.609341 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392263    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:04:44.609341 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392266    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:04:44.609341 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392269    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:04:44.609952 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392271    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:04:44.609952 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392274    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:04:44.609952 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392276    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:04:44.609952 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392279    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:04:44.609952 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392282    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:04:44.609952 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392286    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:04:44.609952 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392290    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:04:44.609952 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392293    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:04:44.609952 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392296    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:04:44.609952 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392299    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:04:44.609952 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392302    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:04:44.609952 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392306    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:04:44.609952 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392311    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:04:44.609952 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392315    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:04:44.609952 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392319    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:04:44.609952 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392322    2569 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:04:44.609952 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392325    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:04:44.609952 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392327    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:04:44.609952 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392330    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:04:44.609952 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392333    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:04:44.823661 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392336    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:04:44.823661 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392338    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:04:44.823661 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392341    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:04:44.823661 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392344    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:04:44.823661 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392347    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:04:44.823661 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392350    2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:04:44.823661 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392354    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:04:44.823661 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392358    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:04:44.823661 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392361    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:04:44.823661 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392364    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:04:44.823661 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392367    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:04:44.823661 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392369    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:04:44.823661 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392372    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:04:44.823661 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392375    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:04:44.823661 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392377    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:04:44.823661 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392381    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:04:44.823661 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392386    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:04:44.823661 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392390    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:04:44.823661 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392394    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:04:44.824392 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392398    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:04:44.824392 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392401    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:04:44.824392 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392403    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:04:44.824392 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392406    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:04:44.824392 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392409    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:04:44.824392 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392412    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:04:44.824392 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392414    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:04:44.824392 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392417    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:04:44.824392 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392419    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:04:44.824392 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.392424    2569 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 13:04:44.824392 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392549    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:04:44.824392 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392557    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:04:44.824392 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392561    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:04:44.824392 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392564    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:04:44.824392 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392567    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:04:44.825548 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392569    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:04:44.825548 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392576    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:04:44.825548 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392579    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:04:44.825548 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392581    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:04:44.825548 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392584    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:04:44.825548 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392588    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:04:44.825548 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392590    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:04:44.825548 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392593    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:04:44.825548 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392596    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:04:44.825548 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392598    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:04:44.825548 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392601    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:04:44.825548 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392604    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:04:44.825548 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392607    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:04:44.825548 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392609    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:04:44.825548 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392611    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:04:44.825548 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392614    2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:04:44.825548 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392617    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:04:44.825548 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392620    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:04:44.825548 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392625    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:04:44.825548 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392629    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:04:45.028135 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392633    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:04:45.028135 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392637    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:04:45.028135 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392639    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:04:45.028135 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392643    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:04:45.028135 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392645    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:04:45.028135 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392648    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:04:45.028135 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392651    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:04:45.028135 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392653    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:04:45.028135 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392656    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:04:45.028135 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392660    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:04:45.028135 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392663    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:04:45.028135 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392665    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:04:45.028135 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392668    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:04:45.028135 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392672    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:04:45.028135 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392675    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:04:45.028135 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392678    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:04:45.028135 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392680    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:04:45.028135 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392683    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:04:45.028135 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392686    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:04:45.028135 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392690    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:04:45.028725 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392694    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:04:45.028725 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392698    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:04:45.028725 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392702    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:04:45.028725 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392706    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:04:45.028725 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392711    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:04:45.028725 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392715    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:04:45.028725 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392718    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:04:45.028725 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392720    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:04:45.028725 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392724    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:04:45.028725 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392728    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:04:45.028725 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392731    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:04:45.028725 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392734    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:04:45.028725 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392736    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:04:45.028725 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392739    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:04:45.028725 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392742    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:04:45.028725 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392745    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:04:45.028725 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392749    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:04:45.028725 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392752    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:04:45.028725 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392755    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:04:45.028725 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392758    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:04:45.029513 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392760    2569 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:04:45.029513 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392767    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:04:45.029513 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392770    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:04:45.029513 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392772    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:04:45.029513 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392775    2569 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:04:45.029513 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392781    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:04:45.029513 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392785    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:04:45.029513 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392790    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:04:45.029513 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392794    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:04:45.029513 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392797    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:04:45.029513 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392799    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:04:45.029513 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392803    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:04:45.029513 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392806    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:04:45.029513 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392808    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:04:45.029513 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392811    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:04:45.029513 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392814    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:04:45.029513 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392816    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:04:45.029513 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392819    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:04:45.029513 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392821    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:04:45.029513 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392824    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:04:45.030044 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:44.392826    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:04:45.030044 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.392832    2569 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 13:04:45.030044 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.393958    2569 server.go:962] "Client rotation is on, will bootstrap in background"
May 25 13:04:45.030044 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.399340    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 25 13:04:45.030044 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.400478    2569 server.go:1019] "Starting client certificate rotation"
May 25 13:04:45.030044 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.400577    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 13:04:45.030044 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.400609    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 13:04:45.030044 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.430273    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 13:04:45.030044 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.432122    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 13:04:45.030044 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.449545    2569 log.go:25] "Validated CRI v1 runtime API"
May 25 13:04:45.030044 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.455892    2569 log.go:25] "Validated CRI v1 image API"
May 25 13:04:45.030044 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.457275    2569 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 25 13:04:45.030044 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.460846    2569 fs.go:135] Filesystem UUIDs: map[1c8bb457-6e5e-49b2-a22f-3889ee4352bb:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9d153129-3234-4eb5-8080-9f7ff18cfe55:/dev/nvme0n1p3]
May 25 13:04:45.030510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.460867    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 25 13:04:45.030510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.464750    2569 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 13:04:45.030716 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.465331    2569 manager.go:217] Machine: {Timestamp:2026-05-25 13:04:44.464776542 +0000 UTC m=+0.467518287 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106954 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21aa47f73f7b6c6d11b4e8e7c73d84 SystemUUID:ec21aa47-f73f-7b6c-6d11-b4e8e7c73d84 BootID:67d81b44-0b9f-471f-9048-ec38df323d52 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e5:e6:19:88:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e5:e6:19:88:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:e6:76:76:50:ea Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 25 13:04:45.030716 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.466068    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 25 13:04:45.030716 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.466158    2569 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 25 13:04:45.030716 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.469972    2569 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 25 13:04:45.030716 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.469997    2569 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-18.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 25 13:04:45.030716 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.470158    2569 topology_manager.go:138] "Creating topology manager with none policy"
May 25 13:04:45.030716 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.470181    2569 container_manager_linux.go:306] "Creating device plugin manager"
May 25 13:04:45.030716 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.470188    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 13:04:45.030716 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.471208    2569 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 13:04:45.030716 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.473203    2569 state_mem.go:36] "Initialized new in-memory state store"
May 25 13:04:45.030716 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.473361    2569 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 25 13:04:45.030716 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.476234    2569 kubelet.go:491] "Attempting to sync node with API server"
May 25 13:04:45.030716 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.476254    2569 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 25 13:04:45.030716 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.476266    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 25 13:04:45.030716 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.476276    2569 kubelet.go:397] "Adding apiserver pod source"
May 25 13:04:45.030716 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.476285    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 25 13:04:45.030716 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.477600    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 13:04:45.030716 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.477617    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.481099    2569 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.483534    2569 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.485772    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.485793    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.485803    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.485812    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.485825    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.485833    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.485841    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.485848    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.485858    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.485865    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.485884    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.485893    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.486899    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.486918    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.490247    2569 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.490289    2569 server.go:1295] "Started kubelet"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:44.490302    2569 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-18.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:44.490306    2569 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.490371    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.490363    2569 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.490442    2569 server_v1.go:47] "podresources" method="list" useActivePods=true
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.491416    2569 server.go:317] "Adding debug handlers to kubelet server"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.492006    2569 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.501093    2569 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-18.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:44.500977    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-18.ec2.internal.18b2d0b660baf0e2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-18.ec2.internal,UID:ip-10-0-139-18.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-18.ec2.internal,},FirstTimestamp:2026-05-25 13:04:44.49025661 +0000 UTC m=+0.492998355,LastTimestamp:2026-05-25 13:04:44.49025661 +0000 UTC m=+0.492998355,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-18.ec2.internal,}"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.502598    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.503297    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.503406    2569 volume_manager.go:295] "The desired_state_of_world populator starts"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.503418    2569 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.503505    2569 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.503541    2569 reconstruct.go:97] "Volume reconstruction finished"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.503549    2569 reconciler.go:26] "Reconciler: start to sync state"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:44.504584    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-18.ec2.internal\" not found"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.508449    2569 factory.go:55] Registering systemd factory
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:44.508461    2569 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.508469    2569 factory.go:223] Registration of the systemd container factory successfully
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.508760    2569 factory.go:153] Registering CRI-O factory
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.508773    2569 factory.go:223] Registration of the crio container factory successfully
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.508835    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 25 13:04:45.031389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.508863    2569 factory.go:103] Registering Raw factory
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.508885    2569 manager.go:1196] Started watching for new ooms in manager
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.510146    2569 manager.go:319] Starting recovery of all containers
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:44.510942    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-18.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:44.510968    2569 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.513233    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6h54l"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.518556    2569 manager.go:324] Recovery completed
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.522358    2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.524303    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6h54l"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.526583    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-18.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.526609    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-18.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.526620    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-18.ec2.internal" event="NodeHasSufficientPID"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.527141    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.527151    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.527183    2569 state_mem.go:36] "Initialized new in-memory state store"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:44.528994    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-18.ec2.internal.18b2d0b662e57172  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-18.ec2.internal,UID:ip-10-0-139-18.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-18.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-18.ec2.internal,},FirstTimestamp:2026-05-25 13:04:44.526596466 +0000 UTC m=+0.529338210,LastTimestamp:2026-05-25 13:04:44.526596466 +0000 UTC m=+0.529338210,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-18.ec2.internal,}"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.531297    2569 policy_none.go:49] "None policy: Start"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.531311    2569 memory_manager.go:186] "Starting memorymanager" policy="None"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.531321    2569 state_mem.go:35] "Initializing new in-memory state store"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.573906    2569 manager.go:341] "Starting Device Plugin manager"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:44.574143    2569 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.574158    2569 server.go:85] "Starting device plugin registration server"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.574442    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.574454    2569 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.574591    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.574683    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.574692    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:44.575225    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:44.575264    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-18.ec2.internal\" not found"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.590623    2569 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.591871    2569 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.591890    2569 status_manager.go:230] "Starting to sync pod status with apiserver"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.591911    2569 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.591920    2569 kubelet.go:2451] "Starting kubelet main sync loop"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:44.591966    2569 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.595905    2569 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.675085    2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.676950    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-18.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.676983    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-18.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.676998    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-18.ec2.internal" event="NodeHasSufficientPID"
May 25 13:04:45.032510 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.677028    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-18.ec2.internal"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.685202    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-18.ec2.internal"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:44.685223    2569 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-18.ec2.internal\": node \"ip-10-0-139-18.ec2.internal\" not found"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.692562    2569 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-18.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-18.ec2.internal"]
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.692626    2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.694493    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-18.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.694519    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-18.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.694529    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-18.ec2.internal" event="NodeHasSufficientPID"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.696833    2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.696959    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-18.ec2.internal"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.697003    2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.697653    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-18.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.697665    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-18.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.697683    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-18.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.697687    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-18.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.697694    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-18.ec2.internal" event="NodeHasSufficientPID"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.697696    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-18.ec2.internal" event="NodeHasSufficientPID"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.699996    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-18.ec2.internal"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.700022    2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.700800    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-18.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.700819    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-18.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.700829    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-18.ec2.internal" event="NodeHasSufficientPID"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:44.703210    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-18.ec2.internal\" not found"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:44.729131    2569 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-18.ec2.internal\" not found" node="ip-10-0-139-18.ec2.internal"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:44.736714    2569 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-18.ec2.internal\" not found" node="ip-10-0-139-18.ec2.internal"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:44.804150    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-18.ec2.internal\" not found"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.805322    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/812b65c6bf3adc1b8ebae49129879464-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-18.ec2.internal\" (UID: \"812b65c6bf3adc1b8ebae49129879464\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-18.ec2.internal"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.805348    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/812b65c6bf3adc1b8ebae49129879464-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-18.ec2.internal\" (UID: \"812b65c6bf3adc1b8ebae49129879464\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-18.ec2.internal"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.805366    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e6365ea56860ef681cb6ea98ee36074a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-18.ec2.internal\" (UID: \"e6365ea56860ef681cb6ea98ee36074a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-18.ec2.internal"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:44.904486    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-18.ec2.internal\" not found"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.905611    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e6365ea56860ef681cb6ea98ee36074a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-18.ec2.internal\" (UID: \"e6365ea56860ef681cb6ea98ee36074a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-18.ec2.internal"
May 25 13:04:45.033544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.905641    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e6365ea56860ef681cb6ea98ee36074a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-18.ec2.internal\" (UID: \"e6365ea56860ef681cb6ea98ee36074a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-18.ec2.internal"
May 25 13:04:45.034350 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.905656    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/812b65c6bf3adc1b8ebae49129879464-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-18.ec2.internal\" (UID: \"812b65c6bf3adc1b8ebae49129879464\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-18.ec2.internal"
May 25 13:04:45.034350 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.905677    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/812b65c6bf3adc1b8ebae49129879464-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-18.ec2.internal\" (UID: \"812b65c6bf3adc1b8ebae49129879464\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-18.ec2.internal"
May 25 13:04:45.034350 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.905706    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/812b65c6bf3adc1b8ebae49129879464-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-18.ec2.internal\" (UID: \"812b65c6bf3adc1b8ebae49129879464\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-18.ec2.internal"
May 25 13:04:45.034350 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:44.905715    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/812b65c6bf3adc1b8ebae49129879464-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-18.ec2.internal\" (UID: \"812b65c6bf3adc1b8ebae49129879464\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-18.ec2.internal"
May 25 13:04:45.034350 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:45.005127    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-18.ec2.internal\" not found"
May 25 13:04:45.034350 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:45.033364    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-18.ec2.internal"
May 25 13:04:45.040654 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:45.040633    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-18.ec2.internal"
May 25 13:04:45.106200 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:45.106101    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-18.ec2.internal\" not found"
May 25 13:04:45.206561 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:45.206527    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-18.ec2.internal\" not found"
May 25 13:04:45.307010 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:45.306977    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-18.ec2.internal\" not found"
May 25 13:04:45.390532 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:45.390456    2569 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:04:45.400459 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:45.400441    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 25 13:04:45.400574 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:45.400557    2569 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 13:04:45.400652 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:45.400617    2569 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 13:04:45.407738 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:45.407709    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-18.ec2.internal\" not found"
May 25 13:04:45.503080 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:45.503056    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 25 13:04:45.508217 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:45.508199    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-18.ec2.internal\" not found"
May 25 13:04:45.518824 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:45.518804    2569 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 13:04:45.526763 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:45.526729    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-24 12:59:44 +0000 UTC" deadline="2027-11-01 11:50:46.347482301 +0000 UTC"
May 25 13:04:45.526763 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:45.526758    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12598h46m0.820727197s"
May 25 13:04:45.539468 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:45.539433    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sdf59"
May 25 13:04:45.547301 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:45.547282    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sdf59"
May 25 13:04:45.608897 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:45.608863    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-18.ec2.internal\" not found"
May 25 13:04:45.617727 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:45.617703    2569 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:04:45.676864 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:45.676808    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod812b65c6bf3adc1b8ebae49129879464.slice/crio-f66cd35ac4e7298508466436f489a7cc4ca88a406fdf64c3b227ffc91ca673f7 WatchSource:0}: Error finding container f66cd35ac4e7298508466436f489a7cc4ca88a406fdf64c3b227ffc91ca673f7: Status 404 returned error can't find the container with id f66cd35ac4e7298508466436f489a7cc4ca88a406fdf64c3b227ffc91ca673f7
May 25 13:04:45.677092 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:45.677073    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6365ea56860ef681cb6ea98ee36074a.slice/crio-f38a0860aa5a29fb3e81e15c78e533ae1f716802dce5ae4093e0febb2bbb8000 WatchSource:0}: Error finding container f38a0860aa5a29fb3e81e15c78e533ae1f716802dce5ae4093e0febb2bbb8000: Status 404 returned error can't find the container with id f38a0860aa5a29fb3e81e15c78e533ae1f716802dce5ae4093e0febb2bbb8000
May 25 13:04:45.681339 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:45.681320    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:04:45.704448 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:45.704414    2569 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-18.ec2.internal"
May 25 13:04:45.721448 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:45.721423    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 13:04:45.722521 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:45.722508    2569 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-18.ec2.internal"
May 25 13:04:45.731244 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:45.731225    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 13:04:46.021583 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.021550    2569 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:04:46.477853 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.477813    2569 apiserver.go:52] "Watching apiserver"
May 25 13:04:46.486682 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.486657    2569 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 25 13:04:46.488577 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.488519    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-z848n","openshift-dns/node-resolver-pz7qk","openshift-image-registry/node-ca-c2xxn","openshift-multus/multus-additional-cni-plugins-x6gsv","openshift-multus/multus-h95n5","openshift-ovn-kubernetes/ovnkube-node-8kgc9","kube-system/kube-apiserver-proxy-ip-10-0-139-18.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-18.ec2.internal","openshift-multus/network-metrics-daemon-8rxxc","openshift-network-diagnostics/network-check-target-5ppwf","openshift-network-operator/iptables-alerter-4lc6w","kube-system/konnectivity-agent-vsr8s"]
May 25 13:04:46.491841 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.491816    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.493971 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.493946    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:04:46.494092 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:46.494033    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppwf" podUID="6a51a8f7-c97c-4de0-b36d-6821814e02d8"
May 25 13:04:46.494383 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.494359    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 25 13:04:46.494477 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.494423    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vkvjl\""
May 25 13:04:46.494539 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.494530    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 25 13:04:46.494648 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.494606    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 25 13:04:46.498492 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.498394    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c2xxn"
May 25 13:04:46.498599 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.498500    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.501215 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.500740    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 25 13:04:46.501215 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.500808    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h95n5"
May 25 13:04:46.501215 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.500983    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 25 13:04:46.501411 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.501229    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 25 13:04:46.501411 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.501374    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 25 13:04:46.501679 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.501661    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 25 13:04:46.501784 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.501693    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 25 13:04:46.501784 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.501710    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 25 13:04:46.501784 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.501663    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k4hbq\""
May 25 13:04:46.501973 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.501871    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 25 13:04:46.502053 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.501980    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2n6lm\""
May 25 13:04:46.503136 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.502977    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 25 13:04:46.503252 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.503135    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kgqpk\""
May 25 13:04:46.505473 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.505456    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.505577 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.505475    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.507689 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.507669    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 25 13:04:46.508005 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.507987    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 25 13:04:46.508097 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.508040    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:04:46.508152 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:46.508097    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rxxc" podUID="8878a1b9-7848-4f10-a0f6-f88059209e87"
May 25 13:04:46.508964 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.508931    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 25 13:04:46.509055 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.508983    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 25 13:04:46.509055 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.509012    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 25 13:04:46.509055 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.509030    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 25 13:04:46.509225 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.508933    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 25 13:04:46.509225 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.508982    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pbglf\""
May 25 13:04:46.509225 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.509156    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hsbxd\""
May 25 13:04:46.509225 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.509194    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 25 13:04:46.510385 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.510361    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pz7qk"
May 25 13:04:46.512536 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.512515    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 25 13:04:46.512856 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.512803    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4lc6w"
May 25 13:04:46.512960 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.512940    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nhngb\""
May 25 13:04:46.513210 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.513186    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 25 13:04:46.513578 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.513558    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-os-release\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.513664 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.513593    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqz2v\" (UniqueName: \"kubernetes.io/projected/3375ebf6-cae9-4704-8051-68847e838a73-kube-api-access-mqz2v\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.513664 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.513619    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07666b34-380e-47af-be65-da7bd0bc3611-cni-binary-copy\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.513664 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.513653    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-run-openvswitch\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.513858 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.513684    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.513858 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.513707    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3375ebf6-cae9-4704-8051-68847e838a73-cni-binary-copy\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.513858 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.513729    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/07666b34-380e-47af-be65-da7bd0bc3611-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.513858 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.513754    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrp9p\" (UniqueName: \"kubernetes.io/projected/07666b34-380e-47af-be65-da7bd0bc3611-kube-api-access-rrp9p\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.513858 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.513778    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f37fe088-db90-44e2-9418-7be91eb347ca-ovnkube-config\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.513858 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.513801    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f37fe088-db90-44e2-9418-7be91eb347ca-ovn-node-metrics-cert\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.513858 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.513824    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzvhq\" (UniqueName: \"kubernetes.io/projected/f37fe088-db90-44e2-9418-7be91eb347ca-kube-api-access-qzvhq\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.513858 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.513846    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6811d512-0b8d-4d99-8615-1bb382d4fdea-device-dir\") pod \"aws-ebs-csi-driver-node-wmqw5\" (UID: \"6811d512-0b8d-4d99-8615-1bb382d4fdea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.514383 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.513870    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-host-run-multus-certs\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.514383 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.513910    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-host-cni-netd\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.514383 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.513932    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f37fe088-db90-44e2-9418-7be91eb347ca-ovnkube-script-lib\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.514383 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.513958    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-etc-systemd\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.514383 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.513979    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-host-run-netns\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.514383 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514018    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/713ea544-1d7a-4451-8439-751bf93048d1-etc-tuned\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.514383 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514071    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-multus-cni-dir\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.514383 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514135    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6811d512-0b8d-4d99-8615-1bb382d4fdea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wmqw5\" (UID: \"6811d512-0b8d-4d99-8615-1bb382d4fdea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.514383 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514182    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4c54f53-586b-496f-bd70-663617ee0a12-host\") pod \"node-ca-c2xxn\" (UID: \"d4c54f53-586b-496f-bd70-663617ee0a12\") " pod="openshift-image-registry/node-ca-c2xxn"
May 25 13:04:46.514383 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514207    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/07666b34-380e-47af-be65-da7bd0bc3611-os-release\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.514383 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514241    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-etc-modprobe-d\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.514383 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514262    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-etc-sysctl-conf\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.514383 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514286    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-system-cni-dir\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.514383 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514324    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-cnibin\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.514383 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514355    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-multus-socket-dir-parent\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.514383 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514383    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-host-var-lib-kubelet\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.515368 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514420    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/07666b34-380e-47af-be65-da7bd0bc3611-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.515368 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514446    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-run-systemd\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.515368 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514470    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-etc-openvswitch\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.515368 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514493    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6811d512-0b8d-4d99-8615-1bb382d4fdea-etc-selinux\") pod \"aws-ebs-csi-driver-node-wmqw5\" (UID: \"6811d512-0b8d-4d99-8615-1bb382d4fdea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.515368 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514516    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07666b34-380e-47af-be65-da7bd0bc3611-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.515368 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514542    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.515368 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514586    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-etc-kubernetes\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.515368 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514623    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-sys\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.515368 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514648    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv4kl\" (UniqueName: \"kubernetes.io/projected/713ea544-1d7a-4451-8439-751bf93048d1-kube-api-access-wv4kl\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.515368 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514680    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-hostroot\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.515368 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514705    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6811d512-0b8d-4d99-8615-1bb382d4fdea-socket-dir\") pod \"aws-ebs-csi-driver-node-wmqw5\" (UID: \"6811d512-0b8d-4d99-8615-1bb382d4fdea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.515368 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514728    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbbrw\" (UniqueName: \"kubernetes.io/projected/d4c54f53-586b-496f-bd70-663617ee0a12-kube-api-access-zbbrw\") pod \"node-ca-c2xxn\" (UID: \"d4c54f53-586b-496f-bd70-663617ee0a12\") " pod="openshift-image-registry/node-ca-c2xxn"
May 25 13:04:46.515368 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514759    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07666b34-380e-47af-be65-da7bd0bc3611-system-cni-dir\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.515368 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514797    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-var-lib-openvswitch\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.515368 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514821    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f37fe088-db90-44e2-9418-7be91eb347ca-env-overrides\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.515368 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514851    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-etc-kubernetes\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.515368 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514900    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 25 13:04:46.516255 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.514958    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-host-run-netns\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.516255 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515035    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-var-lib-kubelet\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.516255 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515088    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6811d512-0b8d-4d99-8615-1bb382d4fdea-registration-dir\") pod \"aws-ebs-csi-driver-node-wmqw5\" (UID: \"6811d512-0b8d-4d99-8615-1bb382d4fdea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.516255 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515118    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6811d512-0b8d-4d99-8615-1bb382d4fdea-sys-fs\") pod \"aws-ebs-csi-driver-node-wmqw5\" (UID: \"6811d512-0b8d-4d99-8615-1bb382d4fdea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.516255 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515201    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-multus-conf-dir\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.516255 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515216    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-s27wz\""
May 25 13:04:46.516255 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515226    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vsr8s"
May 25 13:04:46.516255 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515240    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-run-ovn\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.516255 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515262    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-node-log\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.516255 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515284    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2p7g\" (UniqueName: \"kubernetes.io/projected/6811d512-0b8d-4d99-8615-1bb382d4fdea-kube-api-access-p2p7g\") pod \"aws-ebs-csi-driver-node-wmqw5\" (UID: \"6811d512-0b8d-4d99-8615-1bb382d4fdea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.516255 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515355    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89bs4\" (UniqueName: \"kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4\") pod \"network-check-target-5ppwf\" (UID: \"6a51a8f7-c97c-4de0-b36d-6821814e02d8\") " pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:04:46.516255 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515415    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-host-var-lib-cni-multus\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.516255 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515441    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-etc-sysctl-d\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.516255 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515481    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/713ea544-1d7a-4451-8439-751bf93048d1-tmp\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.516255 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515539    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-host-slash\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.516255 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515541    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 25 13:04:46.516255 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515570    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-host-run-k8s-cni-cncf-io\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.516255 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515582    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 25 13:04:46.516255 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515597    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-host-kubelet\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.518268 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515622    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-host-cni-bin\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.518268 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515645    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-etc-sysconfig\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.518268 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515668    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-lib-modules\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.518268 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515737    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-host-var-lib-cni-bin\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.518268 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515762    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d4c54f53-586b-496f-bd70-663617ee0a12-serviceca\") pod \"node-ca-c2xxn\" (UID: \"d4c54f53-586b-496f-bd70-663617ee0a12\") " pod="openshift-image-registry/node-ca-c2xxn"
May 25 13:04:46.518268 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515786    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07666b34-380e-47af-be65-da7bd0bc3611-cnibin\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.518268 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515810    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3375ebf6-cae9-4704-8051-68847e838a73-multus-daemon-config\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.518268 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515832    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-systemd-units\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.518268 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515855    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-log-socket\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.518268 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515876    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-run\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.518268 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.515915    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-host\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.518268 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.517532    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 25 13:04:46.518268 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.517780    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-d76l6\""
May 25 13:04:46.518268 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.517821    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 25 13:04:46.548042 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.547894    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 12:59:45 +0000 UTC" deadline="2028-02-11 16:13:17.787866328 +0000 UTC"
May 25 13:04:46.548042 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.547927    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15051h8m31.239942592s"
May 25 13:04:46.551435 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.551410    2569 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:04:46.596475 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.596411    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-18.ec2.internal" event={"ID":"812b65c6bf3adc1b8ebae49129879464","Type":"ContainerStarted","Data":"f66cd35ac4e7298508466436f489a7cc4ca88a406fdf64c3b227ffc91ca673f7"}
May 25 13:04:46.597483 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.597455    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-18.ec2.internal" event={"ID":"e6365ea56860ef681cb6ea98ee36074a","Type":"ContainerStarted","Data":"f38a0860aa5a29fb3e81e15c78e533ae1f716802dce5ae4093e0febb2bbb8000"}
May 25 13:04:46.604205 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.604183    2569 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 25 13:04:46.617177 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617127    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bszh4\" (UniqueName: \"kubernetes.io/projected/217de8ce-783f-48b4-bee9-82bbe61f2450-kube-api-access-bszh4\") pod \"node-resolver-pz7qk\" (UID: \"217de8ce-783f-48b4-bee9-82bbe61f2450\") " pod="openshift-dns/node-resolver-pz7qk"
May 25 13:04:46.617324 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617186    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e69ac0d2-7a7a-496b-ba07-c08a6507fd39-konnectivity-ca\") pod \"konnectivity-agent-vsr8s\" (UID: \"e69ac0d2-7a7a-496b-ba07-c08a6507fd39\") " pod="kube-system/konnectivity-agent-vsr8s"
May 25 13:04:46.617324 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617222    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-etc-sysctl-d\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.617324 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617246    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/713ea544-1d7a-4451-8439-751bf93048d1-tmp\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.617324 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617270    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-host-slash\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.617324 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617293    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-host-run-k8s-cni-cncf-io\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.617324 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617317    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-host-kubelet\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.617604 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617341    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-host-cni-bin\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.617604 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617363    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-etc-sysconfig\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.617604 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617373    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-host-slash\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.617604 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617385    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-lib-modules\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.617604 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617436    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-etc-sysctl-d\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.617604 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617444    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-host-var-lib-cni-bin\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.617604 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617471    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-host-var-lib-cni-bin\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.617604 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617491    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d4c54f53-586b-496f-bd70-663617ee0a12-serviceca\") pod \"node-ca-c2xxn\" (UID: \"d4c54f53-586b-496f-bd70-663617ee0a12\") " pod="openshift-image-registry/node-ca-c2xxn"
May 25 13:04:46.617604 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617504    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-host-kubelet\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.617604 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617525    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-lib-modules\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.617604 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617532    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-host-run-k8s-cni-cncf-io\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.617604 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617524    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07666b34-380e-47af-be65-da7bd0bc3611-cnibin\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.617604 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617559    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07666b34-380e-47af-be65-da7bd0bc3611-cnibin\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.617604 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617569    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3375ebf6-cae9-4704-8051-68847e838a73-multus-daemon-config\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.617604 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617599    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/217de8ce-783f-48b4-bee9-82bbe61f2450-tmp-dir\") pod \"node-resolver-pz7qk\" (UID: \"217de8ce-783f-48b4-bee9-82bbe61f2450\") " pod="openshift-dns/node-resolver-pz7qk"
May 25 13:04:46.618326 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617625    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-systemd-units\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.618326 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617649    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-host-cni-bin\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.618326 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617656    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 25 13:04:46.618326 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617672    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-etc-sysconfig\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.618326 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617691    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-log-socket\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.618326 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617653    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-log-socket\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.618326 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617738    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-systemd-units\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.618326 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617778    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-run\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.618326 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617853    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-run\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.618326 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617862    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-host\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.618326 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617897    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-os-release\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.618326 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617923    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mqz2v\" (UniqueName: \"kubernetes.io/projected/3375ebf6-cae9-4704-8051-68847e838a73-kube-api-access-mqz2v\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.618326 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617953    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07666b34-380e-47af-be65-da7bd0bc3611-cni-binary-copy\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.618326 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617956    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d4c54f53-586b-496f-bd70-663617ee0a12-serviceca\") pod \"node-ca-c2xxn\" (UID: \"d4c54f53-586b-496f-bd70-663617ee0a12\") " pod="openshift-image-registry/node-ca-c2xxn"
May 25 13:04:46.618326 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617971    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-host\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.618326 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.617979    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-run-openvswitch\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.618326 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618004    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-os-release\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.618326 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618030    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-run-openvswitch\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.619128 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618186    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.619128 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618217    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3375ebf6-cae9-4704-8051-68847e838a73-cni-binary-copy\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.619128 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618232    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3375ebf6-cae9-4704-8051-68847e838a73-multus-daemon-config\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.619128 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618243    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c9c6100-4de5-4d5a-a25f-9b384efadd19-iptables-alerter-script\") pod \"iptables-alerter-4lc6w\" (UID: \"5c9c6100-4de5-4d5a-a25f-9b384efadd19\") " pod="openshift-network-operator/iptables-alerter-4lc6w"
May 25 13:04:46.619128 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618282    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.619128 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618311    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e69ac0d2-7a7a-496b-ba07-c08a6507fd39-agent-certs\") pod \"konnectivity-agent-vsr8s\" (UID: \"e69ac0d2-7a7a-496b-ba07-c08a6507fd39\") " pod="kube-system/konnectivity-agent-vsr8s"
May 25 13:04:46.619128 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618344    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/07666b34-380e-47af-be65-da7bd0bc3611-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.619128 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618367    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rrp9p\" (UniqueName: \"kubernetes.io/projected/07666b34-380e-47af-be65-da7bd0bc3611-kube-api-access-rrp9p\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.619128 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618388    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f37fe088-db90-44e2-9418-7be91eb347ca-ovnkube-config\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.619128 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618408    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f37fe088-db90-44e2-9418-7be91eb347ca-ovn-node-metrics-cert\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.619128 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618429    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qzvhq\" (UniqueName: \"kubernetes.io/projected/f37fe088-db90-44e2-9418-7be91eb347ca-kube-api-access-qzvhq\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.619128 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618450    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6811d512-0b8d-4d99-8615-1bb382d4fdea-device-dir\") pod \"aws-ebs-csi-driver-node-wmqw5\" (UID: \"6811d512-0b8d-4d99-8615-1bb382d4fdea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.619128 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618470    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-host-run-multus-certs\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.619128 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618490    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-host-cni-netd\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.619128 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618512    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f37fe088-db90-44e2-9418-7be91eb347ca-ovnkube-script-lib\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.619128 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618536    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-etc-systemd\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.619128 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618536    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07666b34-380e-47af-be65-da7bd0bc3611-cni-binary-copy\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.619921 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618583    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-host-run-netns\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.619921 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618607    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs\") pod \"network-metrics-daemon-8rxxc\" (UID: \"8878a1b9-7848-4f10-a0f6-f88059209e87\") " pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:04:46.619921 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618631    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/713ea544-1d7a-4451-8439-751bf93048d1-etc-tuned\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.619921 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618654    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-multus-cni-dir\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.619921 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618677    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6811d512-0b8d-4d99-8615-1bb382d4fdea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wmqw5\" (UID: \"6811d512-0b8d-4d99-8615-1bb382d4fdea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.619921 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618697    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4c54f53-586b-496f-bd70-663617ee0a12-host\") pod \"node-ca-c2xxn\" (UID: \"d4c54f53-586b-496f-bd70-663617ee0a12\") " pod="openshift-image-registry/node-ca-c2xxn"
May 25 13:04:46.619921 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618724    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/07666b34-380e-47af-be65-da7bd0bc3611-os-release\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.619921 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618744    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-etc-modprobe-d\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.619921 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618772    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3375ebf6-cae9-4704-8051-68847e838a73-cni-binary-copy\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.619921 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618767    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-etc-sysctl-conf\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.619921 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618805    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-system-cni-dir\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.619921 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618835    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-cnibin\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.619921 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618856    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-multus-socket-dir-parent\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.619921 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618876    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-host-var-lib-kubelet\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.619921 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618898    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/07666b34-380e-47af-be65-da7bd0bc3611-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.619921 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618917    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-run-systemd\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.619921 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618936    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-etc-openvswitch\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.620731 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618974    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/07666b34-380e-47af-be65-da7bd0bc3611-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.620731 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.618981    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6811d512-0b8d-4d99-8615-1bb382d4fdea-etc-selinux\") pod \"aws-ebs-csi-driver-node-wmqw5\" (UID: \"6811d512-0b8d-4d99-8615-1bb382d4fdea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.620731 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619029    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07666b34-380e-47af-be65-da7bd0bc3611-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.620731 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619067    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.620731 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619080    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6811d512-0b8d-4d99-8615-1bb382d4fdea-etc-selinux\") pod \"aws-ebs-csi-driver-node-wmqw5\" (UID: \"6811d512-0b8d-4d99-8615-1bb382d4fdea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.620731 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619097    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-etc-kubernetes\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.620731 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619121    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-sys\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.620731 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619127    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-multus-cni-dir\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.620731 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619163    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6811d512-0b8d-4d99-8615-1bb382d4fdea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wmqw5\" (UID: \"6811d512-0b8d-4d99-8615-1bb382d4fdea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.620731 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619163    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f37fe088-db90-44e2-9418-7be91eb347ca-ovnkube-config\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.620731 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619227    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-etc-kubernetes\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.620731 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619241    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/07666b34-380e-47af-be65-da7bd0bc3611-os-release\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.620731 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619246    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4c54f53-586b-496f-bd70-663617ee0a12-host\") pod \"node-ca-c2xxn\" (UID: \"d4c54f53-586b-496f-bd70-663617ee0a12\") " pod="openshift-image-registry/node-ca-c2xxn"
May 25 13:04:46.620731 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619259    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wv4kl\" (UniqueName: \"kubernetes.io/projected/713ea544-1d7a-4451-8439-751bf93048d1-kube-api-access-wv4kl\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.620731 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619280    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07666b34-380e-47af-be65-da7bd0bc3611-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.620731 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619291    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-hostroot\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.620731 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619322    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-hostroot\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.621531 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619308    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-etc-modprobe-d\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.621531 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619330    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6811d512-0b8d-4d99-8615-1bb382d4fdea-socket-dir\") pod \"aws-ebs-csi-driver-node-wmqw5\" (UID: \"6811d512-0b8d-4d99-8615-1bb382d4fdea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.621531 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619363    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zbbrw\" (UniqueName: \"kubernetes.io/projected/d4c54f53-586b-496f-bd70-663617ee0a12-kube-api-access-zbbrw\") pod \"node-ca-c2xxn\" (UID: \"d4c54f53-586b-496f-bd70-663617ee0a12\") " pod="openshift-image-registry/node-ca-c2xxn"
May 25 13:04:46.621531 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619371    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-sys\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.621531 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619390    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07666b34-380e-47af-be65-da7bd0bc3611-system-cni-dir\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.621531 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619420    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-var-lib-openvswitch\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.621531 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619443    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f37fe088-db90-44e2-9418-7be91eb347ca-env-overrides\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.621531 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619502    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-etc-kubernetes\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.621531 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619532    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxdkc\" (UniqueName: \"kubernetes.io/projected/8878a1b9-7848-4f10-a0f6-f88059209e87-kube-api-access-xxdkc\") pod \"network-metrics-daemon-8rxxc\" (UID: \"8878a1b9-7848-4f10-a0f6-f88059209e87\") " pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:04:46.621531 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619564    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-host-run-netns\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.621531 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619594    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-var-lib-kubelet\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.621531 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619618    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6811d512-0b8d-4d99-8615-1bb382d4fdea-registration-dir\") pod \"aws-ebs-csi-driver-node-wmqw5\" (UID: \"6811d512-0b8d-4d99-8615-1bb382d4fdea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.621531 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619643    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6811d512-0b8d-4d99-8615-1bb382d4fdea-sys-fs\") pod \"aws-ebs-csi-driver-node-wmqw5\" (UID: \"6811d512-0b8d-4d99-8615-1bb382d4fdea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.621531 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619673    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-multus-conf-dir\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.621531 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619723    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-multus-conf-dir\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.621531 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619767    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-host-run-multus-certs\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.621531 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619809    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c9c6100-4de5-4d5a-a25f-9b384efadd19-host-slash\") pod \"iptables-alerter-4lc6w\" (UID: \"5c9c6100-4de5-4d5a-a25f-9b384efadd19\") " pod="openshift-network-operator/iptables-alerter-4lc6w"
May 25 13:04:46.622310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619845    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47hxk\" (UniqueName: \"kubernetes.io/projected/5c9c6100-4de5-4d5a-a25f-9b384efadd19-kube-api-access-47hxk\") pod \"iptables-alerter-4lc6w\" (UID: \"5c9c6100-4de5-4d5a-a25f-9b384efadd19\") " pod="openshift-network-operator/iptables-alerter-4lc6w"
May 25 13:04:46.622310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619874    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-run-ovn\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.622310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619898    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-node-log\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.622310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619923    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p2p7g\" (UniqueName: \"kubernetes.io/projected/6811d512-0b8d-4d99-8615-1bb382d4fdea-kube-api-access-p2p7g\") pod \"aws-ebs-csi-driver-node-wmqw5\" (UID: \"6811d512-0b8d-4d99-8615-1bb382d4fdea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.622310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619953    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-89bs4\" (UniqueName: \"kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4\") pod \"network-check-target-5ppwf\" (UID: \"6a51a8f7-c97c-4de0-b36d-6821814e02d8\") " pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:04:46.622310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619962    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07666b34-380e-47af-be65-da7bd0bc3611-system-cni-dir\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.622310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619997    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-host-var-lib-cni-multus\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.622310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.620008    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-var-lib-openvswitch\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.622310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619418    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6811d512-0b8d-4d99-8615-1bb382d4fdea-socket-dir\") pod \"aws-ebs-csi-driver-node-wmqw5\" (UID: \"6811d512-0b8d-4d99-8615-1bb382d4fdea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.622310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.620028    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/217de8ce-783f-48b4-bee9-82bbe61f2450-hosts-file\") pod \"node-resolver-pz7qk\" (UID: \"217de8ce-783f-48b4-bee9-82bbe61f2450\") " pod="openshift-dns/node-resolver-pz7qk"
May 25 13:04:46.622310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.620219    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-etc-sysctl-conf\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.622310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.620275    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-system-cni-dir\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.622310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.620292    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-host-run-netns\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.622310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.620320    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-cnibin\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.622310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.620471    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-multus-socket-dir-parent\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.622310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.620519    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-host-var-lib-kubelet\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.622310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.620619    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f37fe088-db90-44e2-9418-7be91eb347ca-ovnkube-script-lib\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.622988 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619122    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-host-cni-netd\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.622988 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.620711    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-host-run-netns\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.622988 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.620739    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-etc-kubernetes\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.622988 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.620782    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6811d512-0b8d-4d99-8615-1bb382d4fdea-registration-dir\") pod \"aws-ebs-csi-driver-node-wmqw5\" (UID: \"6811d512-0b8d-4d99-8615-1bb382d4fdea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.622988 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.620812    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-var-lib-kubelet\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.622988 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.620889    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6811d512-0b8d-4d99-8615-1bb382d4fdea-device-dir\") pod \"aws-ebs-csi-driver-node-wmqw5\" (UID: \"6811d512-0b8d-4d99-8615-1bb382d4fdea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.622988 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.621028    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/713ea544-1d7a-4451-8439-751bf93048d1-etc-systemd\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.622988 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.621063    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/07666b34-380e-47af-be65-da7bd0bc3611-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.622988 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.619187    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.622988 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.621086    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-etc-openvswitch\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.622988 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.621038    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-run-systemd\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.622988 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.621142    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-run-ovn\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.622988 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.621187    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6811d512-0b8d-4d99-8615-1bb382d4fdea-sys-fs\") pod \"aws-ebs-csi-driver-node-wmqw5\" (UID: \"6811d512-0b8d-4d99-8615-1bb382d4fdea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.622988 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.621199    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3375ebf6-cae9-4704-8051-68847e838a73-host-var-lib-cni-multus\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.622988 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.621338    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f37fe088-db90-44e2-9418-7be91eb347ca-node-log\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.622988 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.621799    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f37fe088-db90-44e2-9418-7be91eb347ca-env-overrides\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.622988 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.622003    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f37fe088-db90-44e2-9418-7be91eb347ca-ovn-node-metrics-cert\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.622988 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.622241    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/713ea544-1d7a-4451-8439-751bf93048d1-etc-tuned\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.623765 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.622322    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/713ea544-1d7a-4451-8439-751bf93048d1-tmp\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.628091 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:46.628067    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:04:46.628227 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:46.628095    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:04:46.628227 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:46.628109    2569 projected.go:194] Error preparing data for projected volume kube-api-access-89bs4 for pod openshift-network-diagnostics/network-check-target-5ppwf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:46.628227 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:46.628187    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4 podName:6a51a8f7-c97c-4de0-b36d-6821814e02d8 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:47.128149341 +0000 UTC m=+3.130891078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-89bs4" (UniqueName: "kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4") pod "network-check-target-5ppwf" (UID: "6a51a8f7-c97c-4de0-b36d-6821814e02d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:46.628970 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.628892    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrp9p\" (UniqueName: \"kubernetes.io/projected/07666b34-380e-47af-be65-da7bd0bc3611-kube-api-access-rrp9p\") pod \"multus-additional-cni-plugins-x6gsv\" (UID: \"07666b34-380e-47af-be65-da7bd0bc3611\") " pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.628970 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.628910    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv4kl\" (UniqueName: \"kubernetes.io/projected/713ea544-1d7a-4451-8439-751bf93048d1-kube-api-access-wv4kl\") pod \"tuned-z848n\" (UID: \"713ea544-1d7a-4451-8439-751bf93048d1\") " pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.629298 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.629273    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqz2v\" (UniqueName: \"kubernetes.io/projected/3375ebf6-cae9-4704-8051-68847e838a73-kube-api-access-mqz2v\") pod \"multus-h95n5\" (UID: \"3375ebf6-cae9-4704-8051-68847e838a73\") " pod="openshift-multus/multus-h95n5"
May 25 13:04:46.629762 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.629731    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbbrw\" (UniqueName: \"kubernetes.io/projected/d4c54f53-586b-496f-bd70-663617ee0a12-kube-api-access-zbbrw\") pod \"node-ca-c2xxn\" (UID: \"d4c54f53-586b-496f-bd70-663617ee0a12\") " pod="openshift-image-registry/node-ca-c2xxn"
May 25 13:04:46.630523 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.630486    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzvhq\" (UniqueName: \"kubernetes.io/projected/f37fe088-db90-44e2-9418-7be91eb347ca-kube-api-access-qzvhq\") pod \"ovnkube-node-8kgc9\" (UID: \"f37fe088-db90-44e2-9418-7be91eb347ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.631389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.631369    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2p7g\" (UniqueName: \"kubernetes.io/projected/6811d512-0b8d-4d99-8615-1bb382d4fdea-kube-api-access-p2p7g\") pod \"aws-ebs-csi-driver-node-wmqw5\" (UID: \"6811d512-0b8d-4d99-8615-1bb382d4fdea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.720657 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.720620    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/217de8ce-783f-48b4-bee9-82bbe61f2450-hosts-file\") pod \"node-resolver-pz7qk\" (UID: \"217de8ce-783f-48b4-bee9-82bbe61f2450\") " pod="openshift-dns/node-resolver-pz7qk"
May 25 13:04:46.720871 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.720666    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bszh4\" (UniqueName: \"kubernetes.io/projected/217de8ce-783f-48b4-bee9-82bbe61f2450-kube-api-access-bszh4\") pod \"node-resolver-pz7qk\" (UID: \"217de8ce-783f-48b4-bee9-82bbe61f2450\") " pod="openshift-dns/node-resolver-pz7qk"
May 25 13:04:46.720871 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.720690    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e69ac0d2-7a7a-496b-ba07-c08a6507fd39-konnectivity-ca\") pod \"konnectivity-agent-vsr8s\" (UID: \"e69ac0d2-7a7a-496b-ba07-c08a6507fd39\") " pod="kube-system/konnectivity-agent-vsr8s"
May 25 13:04:46.720871 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.720722    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/217de8ce-783f-48b4-bee9-82bbe61f2450-tmp-dir\") pod \"node-resolver-pz7qk\" (UID: \"217de8ce-783f-48b4-bee9-82bbe61f2450\") " pod="openshift-dns/node-resolver-pz7qk"
May 25 13:04:46.720871 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.720746    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/217de8ce-783f-48b4-bee9-82bbe61f2450-hosts-file\") pod \"node-resolver-pz7qk\" (UID: \"217de8ce-783f-48b4-bee9-82bbe61f2450\") " pod="openshift-dns/node-resolver-pz7qk"
May 25 13:04:46.720871 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.720751    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c9c6100-4de5-4d5a-a25f-9b384efadd19-iptables-alerter-script\") pod \"iptables-alerter-4lc6w\" (UID: \"5c9c6100-4de5-4d5a-a25f-9b384efadd19\") " pod="openshift-network-operator/iptables-alerter-4lc6w"
May 25 13:04:46.720871 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.720828    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e69ac0d2-7a7a-496b-ba07-c08a6507fd39-agent-certs\") pod \"konnectivity-agent-vsr8s\" (UID: \"e69ac0d2-7a7a-496b-ba07-c08a6507fd39\") " pod="kube-system/konnectivity-agent-vsr8s"
May 25 13:04:46.720871 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.720869    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs\") pod \"network-metrics-daemon-8rxxc\" (UID: \"8878a1b9-7848-4f10-a0f6-f88059209e87\") " pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:04:46.721315 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.720918    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xxdkc\" (UniqueName: \"kubernetes.io/projected/8878a1b9-7848-4f10-a0f6-f88059209e87-kube-api-access-xxdkc\") pod \"network-metrics-daemon-8rxxc\" (UID: \"8878a1b9-7848-4f10-a0f6-f88059209e87\") " pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:04:46.721315 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.720947    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c9c6100-4de5-4d5a-a25f-9b384efadd19-host-slash\") pod \"iptables-alerter-4lc6w\" (UID: \"5c9c6100-4de5-4d5a-a25f-9b384efadd19\") " pod="openshift-network-operator/iptables-alerter-4lc6w"
May 25 13:04:46.721315 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.720969    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-47hxk\" (UniqueName: \"kubernetes.io/projected/5c9c6100-4de5-4d5a-a25f-9b384efadd19-kube-api-access-47hxk\") pod \"iptables-alerter-4lc6w\" (UID: \"5c9c6100-4de5-4d5a-a25f-9b384efadd19\") " pod="openshift-network-operator/iptables-alerter-4lc6w"
May 25 13:04:46.721315 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.721059    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c9c6100-4de5-4d5a-a25f-9b384efadd19-host-slash\") pod \"iptables-alerter-4lc6w\" (UID: \"5c9c6100-4de5-4d5a-a25f-9b384efadd19\") " pod="openshift-network-operator/iptables-alerter-4lc6w"
May 25 13:04:46.721315 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.721087    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/217de8ce-783f-48b4-bee9-82bbe61f2450-tmp-dir\") pod \"node-resolver-pz7qk\" (UID: \"217de8ce-783f-48b4-bee9-82bbe61f2450\") " pod="openshift-dns/node-resolver-pz7qk"
May 25 13:04:46.721315 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:46.721208    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:46.721315 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:46.721274    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs podName:8878a1b9-7848-4f10-a0f6-f88059209e87 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:47.221255654 +0000 UTC m=+3.223997401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs") pod "network-metrics-daemon-8rxxc" (UID: "8878a1b9-7848-4f10-a0f6-f88059209e87") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:46.721555 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.721335    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c9c6100-4de5-4d5a-a25f-9b384efadd19-iptables-alerter-script\") pod \"iptables-alerter-4lc6w\" (UID: \"5c9c6100-4de5-4d5a-a25f-9b384efadd19\") " pod="openshift-network-operator/iptables-alerter-4lc6w"
May 25 13:04:46.721555 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.721342    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e69ac0d2-7a7a-496b-ba07-c08a6507fd39-konnectivity-ca\") pod \"konnectivity-agent-vsr8s\" (UID: \"e69ac0d2-7a7a-496b-ba07-c08a6507fd39\") " pod="kube-system/konnectivity-agent-vsr8s"
May 25 13:04:46.723238 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.723222    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e69ac0d2-7a7a-496b-ba07-c08a6507fd39-agent-certs\") pod \"konnectivity-agent-vsr8s\" (UID: \"e69ac0d2-7a7a-496b-ba07-c08a6507fd39\") " pod="kube-system/konnectivity-agent-vsr8s"
May 25 13:04:46.729631 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.729604    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bszh4\" (UniqueName: \"kubernetes.io/projected/217de8ce-783f-48b4-bee9-82bbe61f2450-kube-api-access-bszh4\") pod \"node-resolver-pz7qk\" (UID: \"217de8ce-783f-48b4-bee9-82bbe61f2450\") " pod="openshift-dns/node-resolver-pz7qk"
May 25 13:04:46.729934 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.729917    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47hxk\" (UniqueName: \"kubernetes.io/projected/5c9c6100-4de5-4d5a-a25f-9b384efadd19-kube-api-access-47hxk\") pod \"iptables-alerter-4lc6w\" (UID: \"5c9c6100-4de5-4d5a-a25f-9b384efadd19\") " pod="openshift-network-operator/iptables-alerter-4lc6w"
May 25 13:04:46.730010 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.729929    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxdkc\" (UniqueName: \"kubernetes.io/projected/8878a1b9-7848-4f10-a0f6-f88059209e87-kube-api-access-xxdkc\") pod \"network-metrics-daemon-8rxxc\" (UID: \"8878a1b9-7848-4f10-a0f6-f88059209e87\") " pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:04:46.803014 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.802907    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5"
May 25 13:04:46.812904 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.812873    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c2xxn"
May 25 13:04:46.823762 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.823735    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x6gsv"
May 25 13:04:46.832332 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.832306    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h95n5"
May 25 13:04:46.838957 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.838934    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:04:46.845561 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.845544    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z848n"
May 25 13:04:46.851126 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.851104    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pz7qk"
May 25 13:04:46.859688 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.859667    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4lc6w"
May 25 13:04:46.866248 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:46.866229    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vsr8s"
May 25 13:04:47.223822 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:47.223742    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs\") pod \"network-metrics-daemon-8rxxc\" (UID: \"8878a1b9-7848-4f10-a0f6-f88059209e87\") " pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:04:47.223980 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:47.223833    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-89bs4\" (UniqueName: \"kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4\") pod \"network-check-target-5ppwf\" (UID: \"6a51a8f7-c97c-4de0-b36d-6821814e02d8\") " pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:04:47.223980 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:47.223903    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:47.223980 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:47.223965    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:04:47.224118 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:47.223986    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:04:47.224118 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:47.223999    2569 projected.go:194] Error preparing data for projected volume kube-api-access-89bs4 for pod openshift-network-diagnostics/network-check-target-5ppwf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:47.224118 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:47.223972    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs podName:8878a1b9-7848-4f10-a0f6-f88059209e87 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:48.223955977 +0000 UTC m=+4.226697709 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs") pod "network-metrics-daemon-8rxxc" (UID: "8878a1b9-7848-4f10-a0f6-f88059209e87") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:47.224118 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:47.224063    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4 podName:6a51a8f7-c97c-4de0-b36d-6821814e02d8 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:48.224045507 +0000 UTC m=+4.226787253 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-89bs4" (UniqueName: "kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4") pod "network-check-target-5ppwf" (UID: "6a51a8f7-c97c-4de0-b36d-6821814e02d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:47.256483 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:47.256450    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6811d512_0b8d_4d99_8615_1bb382d4fdea.slice/crio-5d659bc2102b836de866c35f6c153e2936fb3fa904522ec66a8a275f5eac793c WatchSource:0}: Error finding container 5d659bc2102b836de866c35f6c153e2936fb3fa904522ec66a8a275f5eac793c: Status 404 returned error can't find the container with id 5d659bc2102b836de866c35f6c153e2936fb3fa904522ec66a8a275f5eac793c
May 25 13:04:47.258224 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:47.258122    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69ac0d2_7a7a_496b_ba07_c08a6507fd39.slice/crio-b553d0fc5207c59ea9a72fab0c33ef77dc25fde7d731cb65167187edcaf211ab WatchSource:0}: Error finding container b553d0fc5207c59ea9a72fab0c33ef77dc25fde7d731cb65167187edcaf211ab: Status 404 returned error can't find the container with id b553d0fc5207c59ea9a72fab0c33ef77dc25fde7d731cb65167187edcaf211ab
May 25 13:04:47.262387 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:47.262363    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c54f53_586b_496f_bd70_663617ee0a12.slice/crio-bcec01bf2c785d7661a5053ecaf8a952677d3e61a5b967c9a62d89a72a2f811f WatchSource:0}: Error finding container bcec01bf2c785d7661a5053ecaf8a952677d3e61a5b967c9a62d89a72a2f811f: Status 404 returned error can't find the container with id bcec01bf2c785d7661a5053ecaf8a952677d3e61a5b967c9a62d89a72a2f811f
May 25 13:04:47.263113 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:47.263092    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217de8ce_783f_48b4_bee9_82bbe61f2450.slice/crio-1ff3698415e68d7c4941f341c4f048ade06bb21d053b7f273606172105bd6a71 WatchSource:0}: Error finding container 1ff3698415e68d7c4941f341c4f048ade06bb21d053b7f273606172105bd6a71: Status 404 returned error can't find the container with id 1ff3698415e68d7c4941f341c4f048ade06bb21d053b7f273606172105bd6a71
May 25 13:04:47.264136 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:47.264081    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf37fe088_db90_44e2_9418_7be91eb347ca.slice/crio-a30390684d2b08a6b3d31ff90c8dbb416348b9c63577cb0fd2ed50097b4a7a0f WatchSource:0}: Error finding container a30390684d2b08a6b3d31ff90c8dbb416348b9c63577cb0fd2ed50097b4a7a0f: Status 404 returned error can't find the container with id a30390684d2b08a6b3d31ff90c8dbb416348b9c63577cb0fd2ed50097b4a7a0f
May 25 13:04:47.266047 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:47.265917    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c9c6100_4de5_4d5a_a25f_9b384efadd19.slice/crio-a5f0c995f6205e2653e6382a15eabe2e866eaf244b2042096ebdd067622b821e WatchSource:0}: Error finding container a5f0c995f6205e2653e6382a15eabe2e866eaf244b2042096ebdd067622b821e: Status 404 returned error can't find the container with id a5f0c995f6205e2653e6382a15eabe2e866eaf244b2042096ebdd067622b821e
May 25 13:04:47.266993 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:47.266891    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3375ebf6_cae9_4704_8051_68847e838a73.slice/crio-98f312e0504466d732ca53564f53950e41dadc5709c48a6e2013014e2296e26e WatchSource:0}: Error finding container 98f312e0504466d732ca53564f53950e41dadc5709c48a6e2013014e2296e26e: Status 404 returned error can't find the container with id 98f312e0504466d732ca53564f53950e41dadc5709c48a6e2013014e2296e26e
May 25 13:04:47.269458 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:47.269361    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713ea544_1d7a_4451_8439_751bf93048d1.slice/crio-75a001b8379e25766263af78391ca38d84edecaee07b4c9b2a5d90e5139aeb16 WatchSource:0}: Error finding container 75a001b8379e25766263af78391ca38d84edecaee07b4c9b2a5d90e5139aeb16: Status 404 returned error can't find the container with id 75a001b8379e25766263af78391ca38d84edecaee07b4c9b2a5d90e5139aeb16
May 25 13:04:47.269593 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:04:47.269570    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07666b34_380e_47af_be65_da7bd0bc3611.slice/crio-482fe94d3f8326eed408802397f518bcb8c27b6c46b1fdcfa1b5a6bb7f526f98 WatchSource:0}: Error finding container 482fe94d3f8326eed408802397f518bcb8c27b6c46b1fdcfa1b5a6bb7f526f98: Status 404 returned error can't find the container with id 482fe94d3f8326eed408802397f518bcb8c27b6c46b1fdcfa1b5a6bb7f526f98
May 25 13:04:47.548221 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:47.548035    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 12:59:45 +0000 UTC" deadline="2027-11-24 15:30:35.602165996 +0000 UTC"
May 25 13:04:47.548221 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:47.548220    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13154h25m48.053949095s"
May 25 13:04:47.600562 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:47.600531    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6gsv" event={"ID":"07666b34-380e-47af-be65-da7bd0bc3611","Type":"ContainerStarted","Data":"482fe94d3f8326eed408802397f518bcb8c27b6c46b1fdcfa1b5a6bb7f526f98"}
May 25 13:04:47.601556 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:47.601533    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h95n5" event={"ID":"3375ebf6-cae9-4704-8051-68847e838a73","Type":"ContainerStarted","Data":"98f312e0504466d732ca53564f53950e41dadc5709c48a6e2013014e2296e26e"}
May 25 13:04:47.602581 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:47.602558    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9" event={"ID":"f37fe088-db90-44e2-9418-7be91eb347ca","Type":"ContainerStarted","Data":"a30390684d2b08a6b3d31ff90c8dbb416348b9c63577cb0fd2ed50097b4a7a0f"}
May 25 13:04:47.603463 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:47.603443    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pz7qk" event={"ID":"217de8ce-783f-48b4-bee9-82bbe61f2450","Type":"ContainerStarted","Data":"1ff3698415e68d7c4941f341c4f048ade06bb21d053b7f273606172105bd6a71"}
May 25 13:04:47.604322 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:47.604304    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z848n" event={"ID":"713ea544-1d7a-4451-8439-751bf93048d1","Type":"ContainerStarted","Data":"75a001b8379e25766263af78391ca38d84edecaee07b4c9b2a5d90e5139aeb16"}
May 25 13:04:47.605135 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:47.605118    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4lc6w" event={"ID":"5c9c6100-4de5-4d5a-a25f-9b384efadd19","Type":"ContainerStarted","Data":"a5f0c995f6205e2653e6382a15eabe2e866eaf244b2042096ebdd067622b821e"}
May 25 13:04:47.606975 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:47.606957    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c2xxn" event={"ID":"d4c54f53-586b-496f-bd70-663617ee0a12","Type":"ContainerStarted","Data":"bcec01bf2c785d7661a5053ecaf8a952677d3e61a5b967c9a62d89a72a2f811f"}
May 25 13:04:47.607773 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:47.607756    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vsr8s" event={"ID":"e69ac0d2-7a7a-496b-ba07-c08a6507fd39","Type":"ContainerStarted","Data":"b553d0fc5207c59ea9a72fab0c33ef77dc25fde7d731cb65167187edcaf211ab"}
May 25 13:04:47.608694 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:47.608671    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5" event={"ID":"6811d512-0b8d-4d99-8615-1bb382d4fdea","Type":"ContainerStarted","Data":"5d659bc2102b836de866c35f6c153e2936fb3fa904522ec66a8a275f5eac793c"}
May 25 13:04:47.610120 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:47.610091    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-18.ec2.internal" event={"ID":"812b65c6bf3adc1b8ebae49129879464","Type":"ContainerStarted","Data":"262434dffc8e35dc5e726e327cc101abaf5843c7aea5123505e758b2a70a9865"}
May 25 13:04:47.611283 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:47.611260    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-18.ec2.internal" event={"ID":"e6365ea56860ef681cb6ea98ee36074a","Type":"ContainerStarted","Data":"b2ff42cffcc1291206ae69bea693f5ebf71f55762af072bf9fa26722d43774b1"}
May 25 13:04:47.656757 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:47.656695    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-18.ec2.internal" podStartSLOduration=2.656676455 podStartE2EDuration="2.656676455s" podCreationTimestamp="2026-05-25 13:04:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:04:47.655986542 +0000 UTC m=+3.658728296" watchObservedRunningTime="2026-05-25 13:04:47.656676455 +0000 UTC m=+3.659418213"
May 25 13:04:48.091724 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:48.091644    2569 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:04:48.233378 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:48.233342    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs\") pod \"network-metrics-daemon-8rxxc\" (UID: \"8878a1b9-7848-4f10-a0f6-f88059209e87\") " pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:04:48.233566 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:48.233413    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-89bs4\" (UniqueName: \"kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4\") pod \"network-check-target-5ppwf\" (UID: \"6a51a8f7-c97c-4de0-b36d-6821814e02d8\") " pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:04:48.233640 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:48.233610    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:04:48.233640 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:48.233627    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:04:48.233640 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:48.233639    2569 projected.go:194] Error preparing data for projected volume kube-api-access-89bs4 for pod openshift-network-diagnostics/network-check-target-5ppwf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:48.233790 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:48.233703    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4 podName:6a51a8f7-c97c-4de0-b36d-6821814e02d8 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:50.233678371 +0000 UTC m=+6.236420107 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-89bs4" (UniqueName: "kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4") pod "network-check-target-5ppwf" (UID: "6a51a8f7-c97c-4de0-b36d-6821814e02d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:48.234102 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:48.234082    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:48.234193 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:48.234182    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs podName:8878a1b9-7848-4f10-a0f6-f88059209e87 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:50.234120102 +0000 UTC m=+6.236861838 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs") pod "network-metrics-daemon-8rxxc" (UID: "8878a1b9-7848-4f10-a0f6-f88059209e87") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:48.443474 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:48.441767    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-27qqj"]
May 25 13:04:48.447933 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:48.447362    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:04:48.447933 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:48.447494    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27qqj" podUID="e1da9cca-c41c-4afc-955f-46df4c30292a"
May 25 13:04:48.538680 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:48.538640    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e1da9cca-c41c-4afc-955f-46df4c30292a-dbus\") pod \"global-pull-secret-syncer-27qqj\" (UID: \"e1da9cca-c41c-4afc-955f-46df4c30292a\") " pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:04:48.538862 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:48.538711    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e1da9cca-c41c-4afc-955f-46df4c30292a-kubelet-config\") pod \"global-pull-secret-syncer-27qqj\" (UID: \"e1da9cca-c41c-4afc-955f-46df4c30292a\") " pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:04:48.538862 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:48.538763    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e1da9cca-c41c-4afc-955f-46df4c30292a-original-pull-secret\") pod \"global-pull-secret-syncer-27qqj\" (UID: \"e1da9cca-c41c-4afc-955f-46df4c30292a\") " pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:04:48.594526 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:48.594495    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:04:48.594975 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:48.594641    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rxxc" podUID="8878a1b9-7848-4f10-a0f6-f88059209e87"
May 25 13:04:48.595148 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:48.595116    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:04:48.595237 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:48.595223    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppwf" podUID="6a51a8f7-c97c-4de0-b36d-6821814e02d8"
May 25 13:04:48.627252 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:48.626128    2569 generic.go:358] "Generic (PLEG): container finished" podID="812b65c6bf3adc1b8ebae49129879464" containerID="262434dffc8e35dc5e726e327cc101abaf5843c7aea5123505e758b2a70a9865" exitCode=0
May 25 13:04:48.627252 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:48.627142    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-18.ec2.internal" event={"ID":"812b65c6bf3adc1b8ebae49129879464","Type":"ContainerDied","Data":"262434dffc8e35dc5e726e327cc101abaf5843c7aea5123505e758b2a70a9865"}
May 25 13:04:48.627252 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:48.627190    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-18.ec2.internal" event={"ID":"812b65c6bf3adc1b8ebae49129879464","Type":"ContainerStarted","Data":"4b47d9e1446a3efba5ffbfb8f09050808518892c1498d145a8965178c14e5dd0"}
May 25 13:04:48.640079 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:48.640042    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e1da9cca-c41c-4afc-955f-46df4c30292a-original-pull-secret\") pod \"global-pull-secret-syncer-27qqj\" (UID: \"e1da9cca-c41c-4afc-955f-46df4c30292a\") " pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:04:48.640249 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:48.640115    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e1da9cca-c41c-4afc-955f-46df4c30292a-dbus\") pod \"global-pull-secret-syncer-27qqj\" (UID: \"e1da9cca-c41c-4afc-955f-46df4c30292a\") " pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:04:48.640249 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:48.640151    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e1da9cca-c41c-4afc-955f-46df4c30292a-kubelet-config\") pod \"global-pull-secret-syncer-27qqj\" (UID: \"e1da9cca-c41c-4afc-955f-46df4c30292a\") " pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:04:48.640375 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:48.640255    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e1da9cca-c41c-4afc-955f-46df4c30292a-kubelet-config\") pod \"global-pull-secret-syncer-27qqj\" (UID: \"e1da9cca-c41c-4afc-955f-46df4c30292a\") " pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:04:48.640375 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:48.640359    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 13:04:48.640476 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:48.640414    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1da9cca-c41c-4afc-955f-46df4c30292a-original-pull-secret podName:e1da9cca-c41c-4afc-955f-46df4c30292a nodeName:}" failed. No retries permitted until 2026-05-25 13:04:49.14039657 +0000 UTC m=+5.143138315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e1da9cca-c41c-4afc-955f-46df4c30292a-original-pull-secret") pod "global-pull-secret-syncer-27qqj" (UID: "e1da9cca-c41c-4afc-955f-46df4c30292a") : object "kube-system"/"original-pull-secret" not registered
May 25 13:04:48.640748 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:48.640713    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e1da9cca-c41c-4afc-955f-46df4c30292a-dbus\") pod \"global-pull-secret-syncer-27qqj\" (UID: \"e1da9cca-c41c-4afc-955f-46df4c30292a\") " pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:04:49.144625 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:49.144588    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e1da9cca-c41c-4afc-955f-46df4c30292a-original-pull-secret\") pod \"global-pull-secret-syncer-27qqj\" (UID: \"e1da9cca-c41c-4afc-955f-46df4c30292a\") " pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:04:49.144813 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:49.144742    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 13:04:49.144813 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:49.144801    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1da9cca-c41c-4afc-955f-46df4c30292a-original-pull-secret podName:e1da9cca-c41c-4afc-955f-46df4c30292a nodeName:}" failed. No retries permitted until 2026-05-25 13:04:50.144783147 +0000 UTC m=+6.147524900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e1da9cca-c41c-4afc-955f-46df4c30292a-original-pull-secret") pod "global-pull-secret-syncer-27qqj" (UID: "e1da9cca-c41c-4afc-955f-46df4c30292a") : object "kube-system"/"original-pull-secret" not registered
May 25 13:04:50.154236 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:50.154135    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e1da9cca-c41c-4afc-955f-46df4c30292a-original-pull-secret\") pod \"global-pull-secret-syncer-27qqj\" (UID: \"e1da9cca-c41c-4afc-955f-46df4c30292a\") " pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:04:50.154665 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:50.154287    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 13:04:50.154665 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:50.154354    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1da9cca-c41c-4afc-955f-46df4c30292a-original-pull-secret podName:e1da9cca-c41c-4afc-955f-46df4c30292a nodeName:}" failed. No retries permitted until 2026-05-25 13:04:52.154334544 +0000 UTC m=+8.157076286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e1da9cca-c41c-4afc-955f-46df4c30292a-original-pull-secret") pod "global-pull-secret-syncer-27qqj" (UID: "e1da9cca-c41c-4afc-955f-46df4c30292a") : object "kube-system"/"original-pull-secret" not registered
May 25 13:04:50.254838 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:50.254796    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-89bs4\" (UniqueName: \"kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4\") pod \"network-check-target-5ppwf\" (UID: \"6a51a8f7-c97c-4de0-b36d-6821814e02d8\") " pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:04:50.255055 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:50.254896    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs\") pod \"network-metrics-daemon-8rxxc\" (UID: \"8878a1b9-7848-4f10-a0f6-f88059209e87\") " pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:04:50.255055 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:50.255014    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:50.255196 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:50.255072    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs podName:8878a1b9-7848-4f10-a0f6-f88059209e87 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:54.25505519 +0000 UTC m=+10.257796939 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs") pod "network-metrics-daemon-8rxxc" (UID: "8878a1b9-7848-4f10-a0f6-f88059209e87") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:50.255257 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:50.255221    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:04:50.255257 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:50.255235    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:04:50.255257 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:50.255244    2569 projected.go:194] Error preparing data for projected volume kube-api-access-89bs4 for pod openshift-network-diagnostics/network-check-target-5ppwf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:50.255404 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:50.255284    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4 podName:6a51a8f7-c97c-4de0-b36d-6821814e02d8 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:54.255272238 +0000 UTC m=+10.258013985 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-89bs4" (UniqueName: "kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4") pod "network-check-target-5ppwf" (UID: "6a51a8f7-c97c-4de0-b36d-6821814e02d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:50.592513 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:50.592431    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:04:50.592677 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:50.592563    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppwf" podUID="6a51a8f7-c97c-4de0-b36d-6821814e02d8"
May 25 13:04:50.592968 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:50.592950    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:04:50.593080 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:50.593042    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27qqj" podUID="e1da9cca-c41c-4afc-955f-46df4c30292a"
May 25 13:04:50.593142 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:50.593102    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:04:50.593279 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:50.593218    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rxxc" podUID="8878a1b9-7848-4f10-a0f6-f88059209e87"
May 25 13:04:52.172901 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:52.172861    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e1da9cca-c41c-4afc-955f-46df4c30292a-original-pull-secret\") pod \"global-pull-secret-syncer-27qqj\" (UID: \"e1da9cca-c41c-4afc-955f-46df4c30292a\") " pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:04:52.173387 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:52.173058    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 13:04:52.173387 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:52.173119    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1da9cca-c41c-4afc-955f-46df4c30292a-original-pull-secret podName:e1da9cca-c41c-4afc-955f-46df4c30292a nodeName:}" failed. No retries permitted until 2026-05-25 13:04:56.173101424 +0000 UTC m=+12.175843173 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e1da9cca-c41c-4afc-955f-46df4c30292a-original-pull-secret") pod "global-pull-secret-syncer-27qqj" (UID: "e1da9cca-c41c-4afc-955f-46df4c30292a") : object "kube-system"/"original-pull-secret" not registered
May 25 13:04:52.592865 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:52.592788    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:04:52.593018 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:52.592788    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:04:52.593018 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:52.592935    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rxxc" podUID="8878a1b9-7848-4f10-a0f6-f88059209e87"
May 25 13:04:52.593201 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:52.593014    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppwf" podUID="6a51a8f7-c97c-4de0-b36d-6821814e02d8"
May 25 13:04:52.593201 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:52.592788    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:04:52.593201 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:52.593103    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27qqj" podUID="e1da9cca-c41c-4afc-955f-46df4c30292a"
May 25 13:04:54.290655 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:54.290597    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs\") pod \"network-metrics-daemon-8rxxc\" (UID: \"8878a1b9-7848-4f10-a0f6-f88059209e87\") " pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:04:54.291119 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:54.290686    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-89bs4\" (UniqueName: \"kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4\") pod \"network-check-target-5ppwf\" (UID: \"6a51a8f7-c97c-4de0-b36d-6821814e02d8\") " pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:04:54.291119 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:54.290837    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:04:54.291119 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:54.290856    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:04:54.291119 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:54.290868    2569 projected.go:194] Error preparing data for projected volume kube-api-access-89bs4 for pod openshift-network-diagnostics/network-check-target-5ppwf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:54.291119 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:54.290928    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4 podName:6a51a8f7-c97c-4de0-b36d-6821814e02d8 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:02.290909233 +0000 UTC m=+18.293650964 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-89bs4" (UniqueName: "kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4") pod "network-check-target-5ppwf" (UID: "6a51a8f7-c97c-4de0-b36d-6821814e02d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:54.291399 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:54.291243    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:54.291399 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:54.291313    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs podName:8878a1b9-7848-4f10-a0f6-f88059209e87 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:02.291296002 +0000 UTC m=+18.294037745 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs") pod "network-metrics-daemon-8rxxc" (UID: "8878a1b9-7848-4f10-a0f6-f88059209e87") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:54.593700 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:54.593626    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:04:54.593937 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:54.593653    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:04:54.594015 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:54.593681    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:04:54.594363 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:54.594228    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27qqj" podUID="e1da9cca-c41c-4afc-955f-46df4c30292a"
May 25 13:04:54.594363 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:54.594322    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rxxc" podUID="8878a1b9-7848-4f10-a0f6-f88059209e87"
May 25 13:04:54.594902 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:54.594129    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppwf" podUID="6a51a8f7-c97c-4de0-b36d-6821814e02d8"
May 25 13:04:56.205100 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:56.205068    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e1da9cca-c41c-4afc-955f-46df4c30292a-original-pull-secret\") pod \"global-pull-secret-syncer-27qqj\" (UID: \"e1da9cca-c41c-4afc-955f-46df4c30292a\") " pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:04:56.205575 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:56.205266    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 13:04:56.205575 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:56.205352    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1da9cca-c41c-4afc-955f-46df4c30292a-original-pull-secret podName:e1da9cca-c41c-4afc-955f-46df4c30292a nodeName:}" failed. No retries permitted until 2026-05-25 13:05:04.205329982 +0000 UTC m=+20.208071730 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e1da9cca-c41c-4afc-955f-46df4c30292a-original-pull-secret") pod "global-pull-secret-syncer-27qqj" (UID: "e1da9cca-c41c-4afc-955f-46df4c30292a") : object "kube-system"/"original-pull-secret" not registered
May 25 13:04:56.592390 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:56.592299    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:04:56.592390 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:56.592344    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:04:56.592606 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:56.592305    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:04:56.592606 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:56.592448    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppwf" podUID="6a51a8f7-c97c-4de0-b36d-6821814e02d8"
May 25 13:04:56.592606 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:56.592535    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27qqj" podUID="e1da9cca-c41c-4afc-955f-46df4c30292a"
May 25 13:04:56.592756 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:56.592634    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rxxc" podUID="8878a1b9-7848-4f10-a0f6-f88059209e87"
May 25 13:04:58.592429 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:58.592372    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:04:58.592888 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:58.592439    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:04:58.592888 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:58.592521    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppwf" podUID="6a51a8f7-c97c-4de0-b36d-6821814e02d8"
May 25 13:04:58.592888 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:58.592589    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rxxc" podUID="8878a1b9-7848-4f10-a0f6-f88059209e87"
May 25 13:04:58.592888 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:04:58.592638    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:04:58.592888 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:04:58.592752    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27qqj" podUID="e1da9cca-c41c-4afc-955f-46df4c30292a"
May 25 13:05:00.592430 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:00.592304    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:05:00.592430 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:00.592308    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:05:00.592430 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:00.592308    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:05:00.592993 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:00.592491    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27qqj" podUID="e1da9cca-c41c-4afc-955f-46df4c30292a"
May 25 13:05:00.592993 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:00.592579    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rxxc" podUID="8878a1b9-7848-4f10-a0f6-f88059209e87"
May 25 13:05:00.592993 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:00.592653    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppwf" podUID="6a51a8f7-c97c-4de0-b36d-6821814e02d8"
May 25 13:05:02.350691 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:02.350654    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-89bs4\" (UniqueName: \"kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4\") pod \"network-check-target-5ppwf\" (UID: \"6a51a8f7-c97c-4de0-b36d-6821814e02d8\") " pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:05:02.351144 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:02.350732    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs\") pod \"network-metrics-daemon-8rxxc\" (UID: \"8878a1b9-7848-4f10-a0f6-f88059209e87\") " pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:05:02.351144 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:02.350842    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:05:02.351144 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:02.350868    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:05:02.351144 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:02.350877    2569 projected.go:194] Error preparing data for projected volume kube-api-access-89bs4 for pod openshift-network-diagnostics/network-check-target-5ppwf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:02.351144 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:02.350938    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4 podName:6a51a8f7-c97c-4de0-b36d-6821814e02d8 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:18.350918644 +0000 UTC m=+34.353660412 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-89bs4" (UniqueName: "kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4") pod "network-check-target-5ppwf" (UID: "6a51a8f7-c97c-4de0-b36d-6821814e02d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:02.351144 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:02.350849    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:02.351144 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:02.351019    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs podName:8878a1b9-7848-4f10-a0f6-f88059209e87 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:18.350999095 +0000 UTC m=+34.353740841 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs") pod "network-metrics-daemon-8rxxc" (UID: "8878a1b9-7848-4f10-a0f6-f88059209e87") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:02.595458 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:02.595431    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:05:02.595614 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:02.595429    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:05:02.595614 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:02.595544    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27qqj" podUID="e1da9cca-c41c-4afc-955f-46df4c30292a"
May 25 13:05:02.595614 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:02.595429    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:05:02.595776 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:02.595640    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppwf" podUID="6a51a8f7-c97c-4de0-b36d-6821814e02d8"
May 25 13:05:02.595776 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:02.595718    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rxxc" podUID="8878a1b9-7848-4f10-a0f6-f88059209e87"
May 25 13:05:04.264965 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:04.264912    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e1da9cca-c41c-4afc-955f-46df4c30292a-original-pull-secret\") pod \"global-pull-secret-syncer-27qqj\" (UID: \"e1da9cca-c41c-4afc-955f-46df4c30292a\") " pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:05:04.265529 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:04.265091    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 13:05:04.265529 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:04.265197    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1da9cca-c41c-4afc-955f-46df4c30292a-original-pull-secret podName:e1da9cca-c41c-4afc-955f-46df4c30292a nodeName:}" failed. No retries permitted until 2026-05-25 13:05:20.265159488 +0000 UTC m=+36.267901221 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e1da9cca-c41c-4afc-955f-46df4c30292a-original-pull-secret") pod "global-pull-secret-syncer-27qqj" (UID: "e1da9cca-c41c-4afc-955f-46df4c30292a") : object "kube-system"/"original-pull-secret" not registered
May 25 13:05:04.595180 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:04.595138    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:05:04.595352 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:04.595150    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:05:04.595352 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:04.595267    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27qqj" podUID="e1da9cca-c41c-4afc-955f-46df4c30292a"
May 25 13:05:04.595456 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:04.595348    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rxxc" podUID="8878a1b9-7848-4f10-a0f6-f88059209e87"
May 25 13:05:04.595456 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:04.595365    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:05:04.595456 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:04.595446    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppwf" podUID="6a51a8f7-c97c-4de0-b36d-6821814e02d8"
May 25 13:05:05.656300 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:05.656080    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pz7qk" event={"ID":"217de8ce-783f-48b4-bee9-82bbe61f2450","Type":"ContainerStarted","Data":"0a2b41f0583a39d6aab1b5460c7624172c482dc16283e4d313f9ece3cd750a96"}
May 25 13:05:05.657456 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:05.657424    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z848n" event={"ID":"713ea544-1d7a-4451-8439-751bf93048d1","Type":"ContainerStarted","Data":"36b8c766ac3adf2b02df07ff3598be3c197923792ada186f03ecd474c082ada7"}
May 25 13:05:05.658830 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:05.658768    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c2xxn" event={"ID":"d4c54f53-586b-496f-bd70-663617ee0a12","Type":"ContainerStarted","Data":"487793b7a9ebb1c4cf05a3474db5d1b97aff943c350c8715f3699292a67f87ee"}
May 25 13:05:05.661878 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:05.661854    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6gsv" event={"ID":"07666b34-380e-47af-be65-da7bd0bc3611","Type":"ContainerStarted","Data":"194a3b68ad4645e64c604f21b30f04d827353e826c4fbe1bd96d1ef6efff2b43"}
May 25 13:05:05.670914 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:05.670807    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-pz7qk" podStartSLOduration=2.791567434 podStartE2EDuration="20.670777897s" podCreationTimestamp="2026-05-25 13:04:45 +0000 UTC" firstStartedPulling="2026-05-25 13:04:47.264987101 +0000 UTC m=+3.267728848" lastFinishedPulling="2026-05-25 13:05:05.144197575 +0000 UTC m=+21.146939311" observedRunningTime="2026-05-25 13:05:05.670766332 +0000 UTC m=+21.673508089" watchObservedRunningTime="2026-05-25 13:05:05.670777897 +0000 UTC m=+21.673519641"
May 25 13:05:05.671352 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:05.671326    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-18.ec2.internal" podStartSLOduration=20.67131756 podStartE2EDuration="20.67131756s" podCreationTimestamp="2026-05-25 13:04:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:04:48.642532983 +0000 UTC m=+4.645274738" watchObservedRunningTime="2026-05-25 13:05:05.67131756 +0000 UTC m=+21.674059313"
May 25 13:05:05.688593 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:05.688554    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-c2xxn" podStartSLOduration=3.808679592 podStartE2EDuration="21.688539774s" podCreationTimestamp="2026-05-25 13:04:44 +0000 UTC" firstStartedPulling="2026-05-25 13:04:47.264388171 +0000 UTC m=+3.267129916" lastFinishedPulling="2026-05-25 13:05:05.144248363 +0000 UTC m=+21.146990098" observedRunningTime="2026-05-25 13:05:05.688040535 +0000 UTC m=+21.690782302" watchObservedRunningTime="2026-05-25 13:05:05.688539774 +0000 UTC m=+21.691281572"
May 25 13:05:05.739253 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:05.739203    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-z848n" podStartSLOduration=3.639219769 podStartE2EDuration="21.739187848s" podCreationTimestamp="2026-05-25 13:04:44 +0000 UTC" firstStartedPulling="2026-05-25 13:04:47.272486461 +0000 UTC m=+3.275228200" lastFinishedPulling="2026-05-25 13:05:05.372454544 +0000 UTC m=+21.375196279" observedRunningTime="2026-05-25 13:05:05.702937425 +0000 UTC m=+21.705679181" watchObservedRunningTime="2026-05-25 13:05:05.739187848 +0000 UTC m=+21.741929602"
May 25 13:05:06.595490 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:06.595319    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:05:06.595637 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:06.595375    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:05:06.595637 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:06.595569    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27qqj" podUID="e1da9cca-c41c-4afc-955f-46df4c30292a"
May 25 13:05:06.595637 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:06.595393    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:05:06.595637 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:06.595630    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppwf" podUID="6a51a8f7-c97c-4de0-b36d-6821814e02d8"
May 25 13:05:06.595783 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:06.595732    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rxxc" podUID="8878a1b9-7848-4f10-a0f6-f88059209e87"
May 25 13:05:06.665078 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:06.665039    2569 generic.go:358] "Generic (PLEG): container finished" podID="07666b34-380e-47af-be65-da7bd0bc3611" containerID="194a3b68ad4645e64c604f21b30f04d827353e826c4fbe1bd96d1ef6efff2b43" exitCode=0
May 25 13:05:06.665885 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:06.665108    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6gsv" event={"ID":"07666b34-380e-47af-be65-da7bd0bc3611","Type":"ContainerDied","Data":"194a3b68ad4645e64c604f21b30f04d827353e826c4fbe1bd96d1ef6efff2b43"}
May 25 13:05:06.666348 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:06.666318    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h95n5" event={"ID":"3375ebf6-cae9-4704-8051-68847e838a73","Type":"ContainerStarted","Data":"36675fdd4d752df4464b1d8960b2a80eb64cef04afe48fbc8909de32096f6d72"}
May 25 13:05:06.671552 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:06.671533    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:05:06.671855 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:06.671838    2569 generic.go:358] "Generic (PLEG): container finished" podID="f37fe088-db90-44e2-9418-7be91eb347ca" containerID="7d8134430fca014247ce027c8c971c1b1e8ffec66020c65f0bfd719218649b8a" exitCode=1
May 25 13:05:06.671922 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:06.671897    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9" event={"ID":"f37fe088-db90-44e2-9418-7be91eb347ca","Type":"ContainerStarted","Data":"3dd3ba856d7d1bdbf89f7586775df0575605767db30ea818787070115d52f1f5"}
May 25 13:05:06.671922 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:06.671917    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9" event={"ID":"f37fe088-db90-44e2-9418-7be91eb347ca","Type":"ContainerStarted","Data":"8eab470c5627201918158c9a333cc9f2007191c9424e3044ed11f6f6d57958fd"}
May 25 13:05:06.672007 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:06.671927    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9" event={"ID":"f37fe088-db90-44e2-9418-7be91eb347ca","Type":"ContainerStarted","Data":"983168af3ae3cc2cec543b5768877063af90e5489e99888e6633276ac34658f0"}
May 25 13:05:06.672007 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:06.671934    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9" event={"ID":"f37fe088-db90-44e2-9418-7be91eb347ca","Type":"ContainerStarted","Data":"677ed7f4fd0130487d974ef511e5381ab8d2b26bc492ad7f5cb0342c12988436"}
May 25 13:05:06.672007 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:06.671942    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9" event={"ID":"f37fe088-db90-44e2-9418-7be91eb347ca","Type":"ContainerDied","Data":"7d8134430fca014247ce027c8c971c1b1e8ffec66020c65f0bfd719218649b8a"}
May 25 13:05:06.672007 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:06.671951    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9" event={"ID":"f37fe088-db90-44e2-9418-7be91eb347ca","Type":"ContainerStarted","Data":"3b1fde7e3256ee3bac2d5ea2268646f7ddff9498685a7bf3dee4dec42d724cea"}
May 25 13:05:06.673116 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:06.673093    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vsr8s" event={"ID":"e69ac0d2-7a7a-496b-ba07-c08a6507fd39","Type":"ContainerStarted","Data":"9632cbf9200aba9fc2aa5042f7d1c97bbda94693ec400630e8a7785438442597"}
May 25 13:05:06.674288 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:06.674266    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5" event={"ID":"6811d512-0b8d-4d99-8615-1bb382d4fdea","Type":"ContainerStarted","Data":"fa82c57a32c3d04ab0aa11b487e187db64c858ca14171d5493916c0dba2bcbc1"}
May 25 13:05:06.702726 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:06.702643    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-h95n5" podStartSLOduration=4.56869126 podStartE2EDuration="22.702628837s" podCreationTimestamp="2026-05-25 13:04:44 +0000 UTC" firstStartedPulling="2026-05-25 13:04:47.270881702 +0000 UTC m=+3.273623449" lastFinishedPulling="2026-05-25 13:05:05.404819284 +0000 UTC m=+21.407561026" observedRunningTime="2026-05-25 13:05:06.702061941 +0000 UTC m=+22.704803695" watchObservedRunningTime="2026-05-25 13:05:06.702628837 +0000 UTC m=+22.705370591"
May 25 13:05:06.715293 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:06.715251    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vsr8s" podStartSLOduration=3.83146677 podStartE2EDuration="21.715237769s" podCreationTimestamp="2026-05-25 13:04:45 +0000 UTC" firstStartedPulling="2026-05-25 13:04:47.260378671 +0000 UTC m=+3.263120404" lastFinishedPulling="2026-05-25 13:05:05.144149668 +0000 UTC m=+21.146891403" observedRunningTime="2026-05-25 13:05:06.714708471 +0000 UTC m=+22.717450227" watchObservedRunningTime="2026-05-25 13:05:06.715237769 +0000 UTC m=+22.717979522"
May 25 13:05:06.821240 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:06.821219    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 25 13:05:07.587544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:07.587354    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-25T13:05:06.821235346Z","UUID":"17e22fd5-b0bc-4543-a659-ac4cf1fc2a63","Handler":null,"Name":"","Endpoint":""}
May 25 13:05:07.589782 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:07.589720    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 25 13:05:07.589782 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:07.589753    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 25 13:05:07.677809 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:07.677756    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4lc6w" event={"ID":"5c9c6100-4de5-4d5a-a25f-9b384efadd19","Type":"ContainerStarted","Data":"e0b9554730fc36989d7353bcb638545294ca50130f8f9a041535a814bf339d33"}
May 25 13:05:07.679882 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:07.679847    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5" event={"ID":"6811d512-0b8d-4d99-8615-1bb382d4fdea","Type":"ContainerStarted","Data":"5d1b9dd614d3be7bc666619aeaf98d1df26ffdf1ef7ab72e6953fc71bf5f47e8"}
May 25 13:05:07.692246 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:07.692206    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4lc6w" podStartSLOduration=4.555199805 podStartE2EDuration="22.692188955s" podCreationTimestamp="2026-05-25 13:04:45 +0000 UTC" firstStartedPulling="2026-05-25 13:04:47.267873038 +0000 UTC m=+3.270614772" lastFinishedPulling="2026-05-25 13:05:05.404862169 +0000 UTC m=+21.407603922" observedRunningTime="2026-05-25 13:05:07.69183256 +0000 UTC m=+23.694574316" watchObservedRunningTime="2026-05-25 13:05:07.692188955 +0000 UTC m=+23.694930710"
May 25 13:05:08.592417 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:08.592371    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:05:08.592560 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:08.592450    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:05:08.592560 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:08.592469    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:05:08.592631 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:08.592560    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppwf" podUID="6a51a8f7-c97c-4de0-b36d-6821814e02d8"
May 25 13:05:08.592728 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:08.592704    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rxxc" podUID="8878a1b9-7848-4f10-a0f6-f88059209e87"
May 25 13:05:08.592852 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:08.592823    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27qqj" podUID="e1da9cca-c41c-4afc-955f-46df4c30292a"
May 25 13:05:08.683330 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:08.683294    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5" event={"ID":"6811d512-0b8d-4d99-8615-1bb382d4fdea","Type":"ContainerStarted","Data":"035b06e3c7c3ba739f28268c3278807ba957b0f1aa478662a2c085e86e1964dd"}
May 25 13:05:08.685868 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:08.685848    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:05:08.686183 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:08.686142    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9" event={"ID":"f37fe088-db90-44e2-9418-7be91eb347ca","Type":"ContainerStarted","Data":"3faf9cfd71c3e26d6e6c442b4c28e4c568fbdf0c8eae5a66d48a4954cf8d5d57"}
May 25 13:05:08.699715 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:08.699670    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmqw5" podStartSLOduration=4.229352435 podStartE2EDuration="24.699658194s" podCreationTimestamp="2026-05-25 13:04:44 +0000 UTC" firstStartedPulling="2026-05-25 13:04:47.260379914 +0000 UTC m=+3.263121647" lastFinishedPulling="2026-05-25 13:05:07.73068566 +0000 UTC m=+23.733427406" observedRunningTime="2026-05-25 13:05:08.699480344 +0000 UTC m=+24.702222120" watchObservedRunningTime="2026-05-25 13:05:08.699658194 +0000 UTC m=+24.702399948"
May 25 13:05:10.012071 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:10.011791    2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vsr8s"
May 25 13:05:10.012654 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:10.012414    2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vsr8s"
May 25 13:05:10.592825 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:10.592797    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:05:10.592938 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:10.592797    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:05:10.592938 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:10.592912    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:05:10.592938 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:10.592906    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppwf" podUID="6a51a8f7-c97c-4de0-b36d-6821814e02d8"
May 25 13:05:10.593062 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:10.592981    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27qqj" podUID="e1da9cca-c41c-4afc-955f-46df4c30292a"
May 25 13:05:10.593150 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:10.593116    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rxxc" podUID="8878a1b9-7848-4f10-a0f6-f88059209e87"
May 25 13:05:10.695308 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:10.695288    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:05:10.695843 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:10.695695    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9" event={"ID":"f37fe088-db90-44e2-9418-7be91eb347ca","Type":"ContainerStarted","Data":"8a294363dbacb739fa1c732d4828ea0460250dc30d7e6d3540b70ed53dfd1fbf"}
May 25 13:05:10.696338 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:10.696148    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vsr8s"
May 25 13:05:10.696497 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:10.696471    2569 scope.go:117] "RemoveContainer" containerID="7d8134430fca014247ce027c8c971c1b1e8ffec66020c65f0bfd719218649b8a"
May 25 13:05:10.696614 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:10.696590    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vsr8s"
May 25 13:05:11.701442 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:11.701240    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:05:11.702307 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:11.701845    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9" event={"ID":"f37fe088-db90-44e2-9418-7be91eb347ca","Type":"ContainerStarted","Data":"358df8a4faebd5faf17adf9f61463f32b14de4db6a1c5810e486dc3ff8c868e5"}
May 25 13:05:11.702307 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:11.702204    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:05:11.702307 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:11.702241    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:05:11.702307 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:11.702253    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:05:11.720087 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:11.720059    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:05:11.720260 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:11.720203    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:05:11.730959 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:11.730892    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9" podStartSLOduration=9.214930302 podStartE2EDuration="27.73087324s" podCreationTimestamp="2026-05-25 13:04:44 +0000 UTC" firstStartedPulling="2026-05-25 13:04:47.266658148 +0000 UTC m=+3.269399895" lastFinishedPulling="2026-05-25 13:05:05.782601099 +0000 UTC m=+21.785342833" observedRunningTime="2026-05-25 13:05:11.730521164 +0000 UTC m=+27.733263190" watchObservedRunningTime="2026-05-25 13:05:11.73087324 +0000 UTC m=+27.733614994"
May 25 13:05:12.561236 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:12.561139    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-27qqj"]
May 25 13:05:12.561329 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:12.561310    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:05:12.561424 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:12.561402    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27qqj" podUID="e1da9cca-c41c-4afc-955f-46df4c30292a"
May 25 13:05:12.563986 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:12.563961    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5ppwf"]
May 25 13:05:12.564105 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:12.564073    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:05:12.564196 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:12.564160    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppwf" podUID="6a51a8f7-c97c-4de0-b36d-6821814e02d8"
May 25 13:05:12.564577 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:12.564558    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8rxxc"]
May 25 13:05:12.564661 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:12.564650    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:05:12.564743 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:12.564728    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rxxc" podUID="8878a1b9-7848-4f10-a0f6-f88059209e87"
May 25 13:05:12.705588 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:12.705554    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6gsv" event={"ID":"07666b34-380e-47af-be65-da7bd0bc3611","Type":"ContainerStarted","Data":"47f53ef69adfc69c35b0b79bab0dc136d269f048dd47fbc6fb1ab53d4e4a77bf"}
May 25 13:05:13.709586 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:13.709548    2569 generic.go:358] "Generic (PLEG): container finished" podID="07666b34-380e-47af-be65-da7bd0bc3611" containerID="47f53ef69adfc69c35b0b79bab0dc136d269f048dd47fbc6fb1ab53d4e4a77bf" exitCode=0
May 25 13:05:13.710035 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:13.709647    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6gsv" event={"ID":"07666b34-380e-47af-be65-da7bd0bc3611","Type":"ContainerDied","Data":"47f53ef69adfc69c35b0b79bab0dc136d269f048dd47fbc6fb1ab53d4e4a77bf"}
May 25 13:05:14.593945 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:14.593852    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:05:14.594111 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:14.594015    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:05:14.594111 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:14.594054    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:05:14.594111 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:14.594052    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppwf" podUID="6a51a8f7-c97c-4de0-b36d-6821814e02d8"
May 25 13:05:14.594277 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:14.594144    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rxxc" podUID="8878a1b9-7848-4f10-a0f6-f88059209e87"
May 25 13:05:14.594317 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:14.594269    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27qqj" podUID="e1da9cca-c41c-4afc-955f-46df4c30292a"
May 25 13:05:15.714736 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:15.714553    2569 generic.go:358] "Generic (PLEG): container finished" podID="07666b34-380e-47af-be65-da7bd0bc3611" containerID="378a4421d9b277ce8342ae3dee3ef27b0b4c7c875c145874bd6b791f935e42ab" exitCode=0
May 25 13:05:15.714736 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:15.714636    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6gsv" event={"ID":"07666b34-380e-47af-be65-da7bd0bc3611","Type":"ContainerDied","Data":"378a4421d9b277ce8342ae3dee3ef27b0b4c7c875c145874bd6b791f935e42ab"}
May 25 13:05:16.592632 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:16.592595    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:05:16.592783 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:16.592711    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppwf" podUID="6a51a8f7-c97c-4de0-b36d-6821814e02d8"
May 25 13:05:16.592783 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:16.592757    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:05:16.592856 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:16.592845    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27qqj" podUID="e1da9cca-c41c-4afc-955f-46df4c30292a"
May 25 13:05:16.592888 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:16.592869    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:05:16.592935 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:16.592922    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rxxc" podUID="8878a1b9-7848-4f10-a0f6-f88059209e87"
May 25 13:05:17.721334 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:17.721245    2569 generic.go:358] "Generic (PLEG): container finished" podID="07666b34-380e-47af-be65-da7bd0bc3611" containerID="39f01135b3463fa8ea34105e74d1a3c609f3ae3a78ce8497d90a57c59990909e" exitCode=0
May 25 13:05:17.721334 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:17.721297    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6gsv" event={"ID":"07666b34-380e-47af-be65-da7bd0bc3611","Type":"ContainerDied","Data":"39f01135b3463fa8ea34105e74d1a3c609f3ae3a78ce8497d90a57c59990909e"}
May 25 13:05:18.266033 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.265999    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-18.ec2.internal" event="NodeReady"
May 25 13:05:18.266235 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.266159    2569 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 25 13:05:18.308979 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.308899    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84ccf98695-7zpjl"]
May 25 13:05:18.313455 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.313431    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.315931 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.315862    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 25 13:05:18.315931 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.315866    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6tjlh\""
May 25 13:05:18.316144 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.315934    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 25 13:05:18.316144 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.315950    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 25 13:05:18.319264 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.318935    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84ccf98695-7zpjl"]
May 25 13:05:18.319738 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.319674    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-27ppb"]
May 25 13:05:18.322464 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.322444    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 25 13:05:18.324070 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.324052    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7v2wh"]
May 25 13:05:18.324235 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.324219    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-27ppb"
May 25 13:05:18.326577 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.326560    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 25 13:05:18.326668 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.326626    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 25 13:05:18.326939 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.326916    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9wl44\""
May 25 13:05:18.328542 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.328522    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7v2wh"
May 25 13:05:18.331184 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.330721    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zhtj8\""
May 25 13:05:18.331184 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.330730    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 25 13:05:18.331184 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.330745    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 25 13:05:18.331184 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.331026    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 25 13:05:18.331698 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.331680    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7v2wh"]
May 25 13:05:18.334362 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.334181    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-27ppb"]
May 25 13:05:18.371815 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.371779    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs\") pod \"network-metrics-daemon-8rxxc\" (UID: \"8878a1b9-7848-4f10-a0f6-f88059209e87\") " pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:05:18.371996 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.371855    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-89bs4\" (UniqueName: \"kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4\") pod \"network-check-target-5ppwf\" (UID: \"6a51a8f7-c97c-4de0-b36d-6821814e02d8\") " pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:05:18.371996 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:18.371923    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:18.371996 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:18.371957    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:05:18.371996 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:18.371975    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:05:18.371996 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:18.371988    2569 projected.go:194] Error preparing data for projected volume kube-api-access-89bs4 for pod openshift-network-diagnostics/network-check-target-5ppwf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:18.372234 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:18.372003    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs podName:8878a1b9-7848-4f10-a0f6-f88059209e87 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:50.371980415 +0000 UTC m=+66.374722161 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs") pod "network-metrics-daemon-8rxxc" (UID: "8878a1b9-7848-4f10-a0f6-f88059209e87") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:18.372234 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:18.372037    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4 podName:6a51a8f7-c97c-4de0-b36d-6821814e02d8 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:50.37202126 +0000 UTC m=+66.374762998 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-89bs4" (UniqueName: "kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4") pod "network-check-target-5ppwf" (UID: "6a51a8f7-c97c-4de0-b36d-6821814e02d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:18.473296 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.473258    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.473492 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.473308    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4b30c873-1845-484c-ae06-b3eab38d772d-image-registry-private-configuration\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.473492 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.473335    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert\") pod \"ingress-canary-7v2wh\" (UID: \"8b9092df-01f8-4d59-82f8-a058e84b55f5\") " pod="openshift-ingress-canary/ingress-canary-7v2wh"
May 25 13:05:18.473492 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.473361    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kq4z\" (UniqueName: \"kubernetes.io/projected/8b9092df-01f8-4d59-82f8-a058e84b55f5-kube-api-access-9kq4z\") pod \"ingress-canary-7v2wh\" (UID: \"8b9092df-01f8-4d59-82f8-a058e84b55f5\") " pod="openshift-ingress-canary/ingress-canary-7v2wh"
May 25 13:05:18.473492 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.473382    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls\") pod \"dns-default-27ppb\" (UID: \"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293\") " pod="openshift-dns/dns-default-27ppb"
May 25 13:05:18.473492 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.473404    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4b30c873-1845-484c-ae06-b3eab38d772d-ca-trust-extracted\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.473492 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.473454    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b30c873-1845-484c-ae06-b3eab38d772d-trusted-ca\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.473492 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.473479    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz76w\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-kube-api-access-vz76w\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.473838 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.473519    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4b30c873-1845-484c-ae06-b3eab38d772d-installation-pull-secrets\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.473838 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.473557    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-tmp-dir\") pod \"dns-default-27ppb\" (UID: \"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293\") " pod="openshift-dns/dns-default-27ppb"
May 25 13:05:18.473838 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.473578    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctvjj\" (UniqueName: \"kubernetes.io/projected/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-kube-api-access-ctvjj\") pod \"dns-default-27ppb\" (UID: \"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293\") " pod="openshift-dns/dns-default-27ppb"
May 25 13:05:18.473838 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.473602    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-bound-sa-token\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.473838 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.473624    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4b30c873-1845-484c-ae06-b3eab38d772d-registry-certificates\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.473838 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.473713    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-config-volume\") pod \"dns-default-27ppb\" (UID: \"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293\") " pod="openshift-dns/dns-default-27ppb"
May 25 13:05:18.574917 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.574837    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.574917 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.574879    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4b30c873-1845-484c-ae06-b3eab38d772d-image-registry-private-configuration\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.574917 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.574897    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert\") pod \"ingress-canary-7v2wh\" (UID: \"8b9092df-01f8-4d59-82f8-a058e84b55f5\") " pod="openshift-ingress-canary/ingress-canary-7v2wh"
May 25 13:05:18.575222 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.574923    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9kq4z\" (UniqueName: \"kubernetes.io/projected/8b9092df-01f8-4d59-82f8-a058e84b55f5-kube-api-access-9kq4z\") pod \"ingress-canary-7v2wh\" (UID: \"8b9092df-01f8-4d59-82f8-a058e84b55f5\") " pod="openshift-ingress-canary/ingress-canary-7v2wh"
May 25 13:05:18.575222 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.574950    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls\") pod \"dns-default-27ppb\" (UID: \"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293\") " pod="openshift-dns/dns-default-27ppb"
May 25 13:05:18.575222 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:18.575001    2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:05:18.575222 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:18.575013    2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:05:18.575222 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:18.575024    2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84ccf98695-7zpjl: secret "image-registry-tls" not found
May 25 13:05:18.575222 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.575064    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4b30c873-1845-484c-ae06-b3eab38d772d-ca-trust-extracted\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.575222 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:18.575027    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:05:18.575222 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:18.575095    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert podName:8b9092df-01f8-4d59-82f8-a058e84b55f5 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:19.075071696 +0000 UTC m=+35.077813441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert") pod "ingress-canary-7v2wh" (UID: "8b9092df-01f8-4d59-82f8-a058e84b55f5") : secret "canary-serving-cert" not found
May 25 13:05:18.575222 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:18.575118    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls podName:4b30c873-1845-484c-ae06-b3eab38d772d nodeName:}" failed. No retries permitted until 2026-05-25 13:05:19.075108682 +0000 UTC m=+35.077850444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls") pod "image-registry-84ccf98695-7zpjl" (UID: "4b30c873-1845-484c-ae06-b3eab38d772d") : secret "image-registry-tls" not found
May 25 13:05:18.575222 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:18.575135    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls podName:d0b340d4-ffa3-4bf8-aeda-9d13cf88d293 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:19.075126821 +0000 UTC m=+35.077868553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls") pod "dns-default-27ppb" (UID: "d0b340d4-ffa3-4bf8-aeda-9d13cf88d293") : secret "dns-default-metrics-tls" not found
May 25 13:05:18.575222 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.575193    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b30c873-1845-484c-ae06-b3eab38d772d-trusted-ca\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.575222 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.575230    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vz76w\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-kube-api-access-vz76w\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.575864 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.575265    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4b30c873-1845-484c-ae06-b3eab38d772d-installation-pull-secrets\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.575864 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.575318    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-tmp-dir\") pod \"dns-default-27ppb\" (UID: \"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293\") " pod="openshift-dns/dns-default-27ppb"
May 25 13:05:18.575864 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.575343    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ctvjj\" (UniqueName: \"kubernetes.io/projected/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-kube-api-access-ctvjj\") pod \"dns-default-27ppb\" (UID: \"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293\") " pod="openshift-dns/dns-default-27ppb"
May 25 13:05:18.575864 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.575480    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4b30c873-1845-484c-ae06-b3eab38d772d-ca-trust-extracted\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.575864 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.575567    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-bound-sa-token\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.575864 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.575609    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4b30c873-1845-484c-ae06-b3eab38d772d-registry-certificates\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.575864 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.575642    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-config-volume\") pod \"dns-default-27ppb\" (UID: \"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293\") " pod="openshift-dns/dns-default-27ppb"
May 25 13:05:18.575864 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.575723    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-tmp-dir\") pod \"dns-default-27ppb\" (UID: \"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293\") " pod="openshift-dns/dns-default-27ppb"
May 25 13:05:18.576196 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.576117    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-config-volume\") pod \"dns-default-27ppb\" (UID: \"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293\") " pod="openshift-dns/dns-default-27ppb"
May 25 13:05:18.576196 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.576151    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4b30c873-1845-484c-ae06-b3eab38d772d-registry-certificates\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.576293 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.576206    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b30c873-1845-484c-ae06-b3eab38d772d-trusted-ca\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.579629 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.579606    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4b30c873-1845-484c-ae06-b3eab38d772d-installation-pull-secrets\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.579758 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.579635    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4b30c873-1845-484c-ae06-b3eab38d772d-image-registry-private-configuration\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.584412 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.584359    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-bound-sa-token\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.584563 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.584438    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctvjj\" (UniqueName: \"kubernetes.io/projected/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-kube-api-access-ctvjj\") pod \"dns-default-27ppb\" (UID: \"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293\") " pod="openshift-dns/dns-default-27ppb"
May 25 13:05:18.584800 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.584778    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kq4z\" (UniqueName: \"kubernetes.io/projected/8b9092df-01f8-4d59-82f8-a058e84b55f5-kube-api-access-9kq4z\") pod \"ingress-canary-7v2wh\" (UID: \"8b9092df-01f8-4d59-82f8-a058e84b55f5\") " pod="openshift-ingress-canary/ingress-canary-7v2wh"
May 25 13:05:18.584912 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.584890    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz76w\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-kube-api-access-vz76w\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:18.592268 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.592236    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:05:18.592992 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.592971    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:05:18.593772 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.593747    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:05:18.594746 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.594723    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rx2c5\""
May 25 13:05:18.594889 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.594869    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 13:05:18.595118 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.595104    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cw8xk\""
May 25 13:05:18.595210 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.595139    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 13:05:18.595345 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.595329    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 13:05:18.595643 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:18.595629    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 25 13:05:19.081493 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:19.081459    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:19.082214 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:19.081500    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert\") pod \"ingress-canary-7v2wh\" (UID: \"8b9092df-01f8-4d59-82f8-a058e84b55f5\") " pod="openshift-ingress-canary/ingress-canary-7v2wh"
May 25 13:05:19.082214 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:19.081642    2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:05:19.082214 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:19.081667    2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84ccf98695-7zpjl: secret "image-registry-tls" not found
May 25 13:05:19.082214 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:19.081732    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls podName:4b30c873-1845-484c-ae06-b3eab38d772d nodeName:}" failed. No retries permitted until 2026-05-25 13:05:20.081712469 +0000 UTC m=+36.084454205 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls") pod "image-registry-84ccf98695-7zpjl" (UID: "4b30c873-1845-484c-ae06-b3eab38d772d") : secret "image-registry-tls" not found
May 25 13:05:19.082214 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:19.081643    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls\") pod \"dns-default-27ppb\" (UID: \"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293\") " pod="openshift-dns/dns-default-27ppb"
May 25 13:05:19.082214 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:19.081782    2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:05:19.082214 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:19.081849    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert podName:8b9092df-01f8-4d59-82f8-a058e84b55f5 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:20.081832613 +0000 UTC m=+36.084574347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert") pod "ingress-canary-7v2wh" (UID: "8b9092df-01f8-4d59-82f8-a058e84b55f5") : secret "canary-serving-cert" not found
May 25 13:05:19.082214 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:19.081869    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:05:19.082214 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:19.081965    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls podName:d0b340d4-ffa3-4bf8-aeda-9d13cf88d293 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:20.081950761 +0000 UTC m=+36.084692495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls") pod "dns-default-27ppb" (UID: "d0b340d4-ffa3-4bf8-aeda-9d13cf88d293") : secret "dns-default-metrics-tls" not found
May 25 13:05:20.091344 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:20.091298    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:20.091790 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:20.091353    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert\") pod \"ingress-canary-7v2wh\" (UID: \"8b9092df-01f8-4d59-82f8-a058e84b55f5\") " pod="openshift-ingress-canary/ingress-canary-7v2wh"
May 25 13:05:20.091790 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:20.091381    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls\") pod \"dns-default-27ppb\" (UID: \"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293\") " pod="openshift-dns/dns-default-27ppb"
May 25 13:05:20.091790 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:20.091455    2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:05:20.091790 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:20.091479    2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84ccf98695-7zpjl: secret "image-registry-tls" not found
May 25 13:05:20.091790 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:20.091492    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:05:20.091790 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:20.091509    2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:05:20.091790 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:20.091539    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls podName:4b30c873-1845-484c-ae06-b3eab38d772d nodeName:}" failed. No retries permitted until 2026-05-25 13:05:22.091524564 +0000 UTC m=+38.094266297 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls") pod "image-registry-84ccf98695-7zpjl" (UID: "4b30c873-1845-484c-ae06-b3eab38d772d") : secret "image-registry-tls" not found
May 25 13:05:20.091790 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:20.091562    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert podName:8b9092df-01f8-4d59-82f8-a058e84b55f5 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:22.091545073 +0000 UTC m=+38.094286823 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert") pod "ingress-canary-7v2wh" (UID: "8b9092df-01f8-4d59-82f8-a058e84b55f5") : secret "canary-serving-cert" not found
May 25 13:05:20.091790 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:20.091580    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls podName:d0b340d4-ffa3-4bf8-aeda-9d13cf88d293 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:22.091571389 +0000 UTC m=+38.094313124 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls") pod "dns-default-27ppb" (UID: "d0b340d4-ffa3-4bf8-aeda-9d13cf88d293") : secret "dns-default-metrics-tls" not found
May 25 13:05:20.293298 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:20.293259    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e1da9cca-c41c-4afc-955f-46df4c30292a-original-pull-secret\") pod \"global-pull-secret-syncer-27qqj\" (UID: \"e1da9cca-c41c-4afc-955f-46df4c30292a\") " pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:05:20.296087 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:20.296030    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e1da9cca-c41c-4afc-955f-46df4c30292a-original-pull-secret\") pod \"global-pull-secret-syncer-27qqj\" (UID: \"e1da9cca-c41c-4afc-955f-46df4c30292a\") " pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:05:20.419282 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:20.419249    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27qqj"
May 25 13:05:20.574610 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:20.574274    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-27qqj"]
May 25 13:05:20.579192 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:05:20.579143    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1da9cca_c41c_4afc_955f_46df4c30292a.slice/crio-369a483861c3120df61004fe6bb47ff4a3d10772489a402b6e25700f7c1d2327 WatchSource:0}: Error finding container 369a483861c3120df61004fe6bb47ff4a3d10772489a402b6e25700f7c1d2327: Status 404 returned error can't find the container with id 369a483861c3120df61004fe6bb47ff4a3d10772489a402b6e25700f7c1d2327
May 25 13:05:20.729128 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:20.729012    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-27qqj" event={"ID":"e1da9cca-c41c-4afc-955f-46df4c30292a","Type":"ContainerStarted","Data":"369a483861c3120df61004fe6bb47ff4a3d10772489a402b6e25700f7c1d2327"}
May 25 13:05:22.109606 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:22.109555    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:22.109606 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:22.109605    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert\") pod \"ingress-canary-7v2wh\" (UID: \"8b9092df-01f8-4d59-82f8-a058e84b55f5\") " pod="openshift-ingress-canary/ingress-canary-7v2wh"
May 25 13:05:22.110127 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:22.109652    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls\") pod \"dns-default-27ppb\" (UID: \"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293\") " pod="openshift-dns/dns-default-27ppb"
May 25 13:05:22.110127 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:22.109734    2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:05:22.110127 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:22.109756    2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84ccf98695-7zpjl: secret "image-registry-tls" not found
May 25 13:05:22.110127 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:22.109799    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:05:22.110127 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:22.109827    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls podName:4b30c873-1845-484c-ae06-b3eab38d772d nodeName:}" failed. No retries permitted until 2026-05-25 13:05:26.109807203 +0000 UTC m=+42.112548948 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls") pod "image-registry-84ccf98695-7zpjl" (UID: "4b30c873-1845-484c-ae06-b3eab38d772d") : secret "image-registry-tls" not found
May 25 13:05:22.110127 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:22.109850    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls podName:d0b340d4-ffa3-4bf8-aeda-9d13cf88d293 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:26.109838179 +0000 UTC m=+42.112579930 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls") pod "dns-default-27ppb" (UID: "d0b340d4-ffa3-4bf8-aeda-9d13cf88d293") : secret "dns-default-metrics-tls" not found
May 25 13:05:22.110127 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:22.109900    2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:05:22.110127 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:22.109931    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert podName:8b9092df-01f8-4d59-82f8-a058e84b55f5 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:26.109924258 +0000 UTC m=+42.112665990 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert") pod "ingress-canary-7v2wh" (UID: "8b9092df-01f8-4d59-82f8-a058e84b55f5") : secret "canary-serving-cert" not found
May 25 13:05:25.741074 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:25.740982    2569 generic.go:358] "Generic (PLEG): container finished" podID="07666b34-380e-47af-be65-da7bd0bc3611" containerID="e234f9d6963ac4004f898a3e8c6e901cae05fc13b1da5e21e93bb18fbb07e408" exitCode=0
May 25 13:05:25.741074 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:25.741022    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6gsv" event={"ID":"07666b34-380e-47af-be65-da7bd0bc3611","Type":"ContainerDied","Data":"e234f9d6963ac4004f898a3e8c6e901cae05fc13b1da5e21e93bb18fbb07e408"}
May 25 13:05:25.742334 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:25.742313    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-27qqj" event={"ID":"e1da9cca-c41c-4afc-955f-46df4c30292a","Type":"ContainerStarted","Data":"682e160d819e8225feb1551f61b39d431cb2a5220abd124672cc77adbfc31453"}
May 25 13:05:25.779415 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:25.779355    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-27qqj" podStartSLOduration=33.28349964 podStartE2EDuration="37.779333411s" podCreationTimestamp="2026-05-25 13:04:48 +0000 UTC" firstStartedPulling="2026-05-25 13:05:20.581018131 +0000 UTC m=+36.583759863" lastFinishedPulling="2026-05-25 13:05:25.076851903 +0000 UTC m=+41.079593634" observedRunningTime="2026-05-25 13:05:25.778448414 +0000 UTC m=+41.781190169" watchObservedRunningTime="2026-05-25 13:05:25.779333411 +0000 UTC m=+41.782075165"
May 25 13:05:26.138050 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:26.138012    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:26.138050 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:26.138057    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert\") pod \"ingress-canary-7v2wh\" (UID: \"8b9092df-01f8-4d59-82f8-a058e84b55f5\") " pod="openshift-ingress-canary/ingress-canary-7v2wh"
May 25 13:05:26.138339 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:26.138081    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls\") pod \"dns-default-27ppb\" (UID: \"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293\") " pod="openshift-dns/dns-default-27ppb"
May 25 13:05:26.138339 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:26.138188    2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:05:26.138339 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:26.138197    2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:05:26.138339 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:26.138216    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:05:26.138339 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:26.138221    2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84ccf98695-7zpjl: secret "image-registry-tls" not found
May 25 13:05:26.138339 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:26.138255    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert podName:8b9092df-01f8-4d59-82f8-a058e84b55f5 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:34.138239005 +0000 UTC m=+50.140980736 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert") pod "ingress-canary-7v2wh" (UID: "8b9092df-01f8-4d59-82f8-a058e84b55f5") : secret "canary-serving-cert" not found
May 25 13:05:26.138339 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:26.138269    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls podName:d0b340d4-ffa3-4bf8-aeda-9d13cf88d293 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:34.138262108 +0000 UTC m=+50.141003839 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls") pod "dns-default-27ppb" (UID: "d0b340d4-ffa3-4bf8-aeda-9d13cf88d293") : secret "dns-default-metrics-tls" not found
May 25 13:05:26.138339 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:26.138280    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls podName:4b30c873-1845-484c-ae06-b3eab38d772d nodeName:}" failed. No retries permitted until 2026-05-25 13:05:34.138274621 +0000 UTC m=+50.141016352 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls") pod "image-registry-84ccf98695-7zpjl" (UID: "4b30c873-1845-484c-ae06-b3eab38d772d") : secret "image-registry-tls" not found
May 25 13:05:26.747815 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:26.747782    2569 generic.go:358] "Generic (PLEG): container finished" podID="07666b34-380e-47af-be65-da7bd0bc3611" containerID="46ebd12ed9385da33b54df5b4948cdcbf7cdef06a02c1009894b2c611039af38" exitCode=0
May 25 13:05:26.748248 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:26.747852    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6gsv" event={"ID":"07666b34-380e-47af-be65-da7bd0bc3611","Type":"ContainerDied","Data":"46ebd12ed9385da33b54df5b4948cdcbf7cdef06a02c1009894b2c611039af38"}
May 25 13:05:27.752620 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:27.752583    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6gsv" event={"ID":"07666b34-380e-47af-be65-da7bd0bc3611","Type":"ContainerStarted","Data":"3fc19014dbe9da415e56071d9dd047cedb6b6bada4d96fba56267ada893ef9bb"}
May 25 13:05:27.776818 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:27.776769    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x6gsv" podStartSLOduration=5.97275257 podStartE2EDuration="43.776755097s" podCreationTimestamp="2026-05-25 13:04:44 +0000 UTC" firstStartedPulling="2026-05-25 13:04:47.272494643 +0000 UTC m=+3.275236389" lastFinishedPulling="2026-05-25 13:05:25.076497184 +0000 UTC m=+41.079238916" observedRunningTime="2026-05-25 13:05:27.77527921 +0000 UTC m=+43.778020964" watchObservedRunningTime="2026-05-25 13:05:27.776755097 +0000 UTC m=+43.779496850"
May 25 13:05:34.194820 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:34.194777    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:34.194820 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:34.194818    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert\") pod \"ingress-canary-7v2wh\" (UID: \"8b9092df-01f8-4d59-82f8-a058e84b55f5\") " pod="openshift-ingress-canary/ingress-canary-7v2wh"
May 25 13:05:34.195331 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:34.194836    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls\") pod \"dns-default-27ppb\" (UID: \"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293\") " pod="openshift-dns/dns-default-27ppb"
May 25 13:05:34.195331 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:34.194931    2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:05:34.195331 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:34.194996    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert podName:8b9092df-01f8-4d59-82f8-a058e84b55f5 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:50.194981464 +0000 UTC m=+66.197723196 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert") pod "ingress-canary-7v2wh" (UID: "8b9092df-01f8-4d59-82f8-a058e84b55f5") : secret "canary-serving-cert" not found
May 25 13:05:34.195331 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:34.194932    2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:05:34.195331 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:34.195035    2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84ccf98695-7zpjl: secret "image-registry-tls" not found
May 25 13:05:34.195331 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:34.195084    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls podName:4b30c873-1845-484c-ae06-b3eab38d772d nodeName:}" failed. No retries permitted until 2026-05-25 13:05:50.195073325 +0000 UTC m=+66.197815057 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls") pod "image-registry-84ccf98695-7zpjl" (UID: "4b30c873-1845-484c-ae06-b3eab38d772d") : secret "image-registry-tls" not found
May 25 13:05:34.195331 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:34.194940    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:05:34.195331 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:34.195117    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls podName:d0b340d4-ffa3-4bf8-aeda-9d13cf88d293 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:50.195108932 +0000 UTC m=+66.197850664 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls") pod "dns-default-27ppb" (UID: "d0b340d4-ffa3-4bf8-aeda-9d13cf88d293") : secret "dns-default-metrics-tls" not found
May 25 13:05:43.720602 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:43.720565    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kgc9"
May 25 13:05:50.200762 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:50.200702    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:05:50.200762 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:50.200769    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert\") pod \"ingress-canary-7v2wh\" (UID: \"8b9092df-01f8-4d59-82f8-a058e84b55f5\") " pod="openshift-ingress-canary/ingress-canary-7v2wh"
May 25 13:05:50.201239 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:50.200802    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls\") pod \"dns-default-27ppb\" (UID: \"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293\") " pod="openshift-dns/dns-default-27ppb"
May 25 13:05:50.201239 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:50.200860    2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:05:50.201239 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:50.200908    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:05:50.201239 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:50.200934    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert podName:8b9092df-01f8-4d59-82f8-a058e84b55f5 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:22.200919391 +0000 UTC m=+98.203661127 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert") pod "ingress-canary-7v2wh" (UID: "8b9092df-01f8-4d59-82f8-a058e84b55f5") : secret "canary-serving-cert" not found
May 25 13:05:50.201239 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:50.200864    2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:05:50.201239 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:50.200962    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls podName:d0b340d4-ffa3-4bf8-aeda-9d13cf88d293 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:22.200944719 +0000 UTC m=+98.203686461 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls") pod "dns-default-27ppb" (UID: "d0b340d4-ffa3-4bf8-aeda-9d13cf88d293") : secret "dns-default-metrics-tls" not found
May 25 13:05:50.201239 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:50.200968    2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84ccf98695-7zpjl: secret "image-registry-tls" not found
May 25 13:05:50.201239 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:50.201012    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls podName:4b30c873-1845-484c-ae06-b3eab38d772d nodeName:}" failed. No retries permitted until 2026-05-25 13:06:22.200998234 +0000 UTC m=+98.203739987 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls") pod "image-registry-84ccf98695-7zpjl" (UID: "4b30c873-1845-484c-ae06-b3eab38d772d") : secret "image-registry-tls" not found
May 25 13:05:50.402004 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:50.401960    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs\") pod \"network-metrics-daemon-8rxxc\" (UID: \"8878a1b9-7848-4f10-a0f6-f88059209e87\") " pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:05:50.402224 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:50.402030    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-89bs4\" (UniqueName: \"kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4\") pod \"network-check-target-5ppwf\" (UID: \"6a51a8f7-c97c-4de0-b36d-6821814e02d8\") " pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:05:50.404657 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:50.404626    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 13:05:50.404809 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:50.404723    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 13:05:50.412555 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:50.412526    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 25 13:05:50.412667 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:05:50.412606    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs podName:8878a1b9-7848-4f10-a0f6-f88059209e87 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:54.412589364 +0000 UTC m=+130.415331096 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs") pod "network-metrics-daemon-8rxxc" (UID: "8878a1b9-7848-4f10-a0f6-f88059209e87") : secret "metrics-daemon-secret" not found
May 25 13:05:50.416033 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:50.416001    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 13:05:50.426389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:50.426358    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89bs4\" (UniqueName: \"kubernetes.io/projected/6a51a8f7-c97c-4de0-b36d-6821814e02d8-kube-api-access-89bs4\") pod \"network-check-target-5ppwf\" (UID: \"6a51a8f7-c97c-4de0-b36d-6821814e02d8\") " pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:05:50.714989 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:50.714955    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cw8xk\""
May 25 13:05:50.722594 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:50.722571    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:05:50.844857 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:50.844823    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5ppwf"]
May 25 13:05:50.848432 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:05:50.848404    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a51a8f7_c97c_4de0_b36d_6821814e02d8.slice/crio-e66428454013967d9b4d23f1264274c4ede8082eb2e4c85eb440e6180b4c3aea WatchSource:0}: Error finding container e66428454013967d9b4d23f1264274c4ede8082eb2e4c85eb440e6180b4c3aea: Status 404 returned error can't find the container with id e66428454013967d9b4d23f1264274c4ede8082eb2e4c85eb440e6180b4c3aea
May 25 13:05:51.797660 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:51.797623    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5ppwf" event={"ID":"6a51a8f7-c97c-4de0-b36d-6821814e02d8","Type":"ContainerStarted","Data":"e66428454013967d9b4d23f1264274c4ede8082eb2e4c85eb440e6180b4c3aea"}
May 25 13:05:54.804252 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:54.804212    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5ppwf" event={"ID":"6a51a8f7-c97c-4de0-b36d-6821814e02d8","Type":"ContainerStarted","Data":"34971779f40f4d00187bd0031cb587228e2fb7027fdc5a7c0f82f47636f3722f"}
May 25 13:05:54.804620 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:54.804314    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:05:54.819392 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:05:54.819345    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5ppwf" podStartSLOduration=67.631032039 podStartE2EDuration="1m10.819328388s" podCreationTimestamp="2026-05-25 13:04:44 +0000 UTC" firstStartedPulling="2026-05-25 13:05:50.850213895 +0000 UTC m=+66.852955631" lastFinishedPulling="2026-05-25 13:05:54.038510245 +0000 UTC m=+70.041251980" observedRunningTime="2026-05-25 13:05:54.818836932 +0000 UTC m=+70.821578687" watchObservedRunningTime="2026-05-25 13:05:54.819328388 +0000 UTC m=+70.822070139"
May 25 13:06:22.231342 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:22.231295    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls\") pod \"dns-default-27ppb\" (UID: \"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293\") " pod="openshift-dns/dns-default-27ppb"
May 25 13:06:22.231729 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:22.231383    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:06:22.231729 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:22.231405    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert\") pod \"ingress-canary-7v2wh\" (UID: \"8b9092df-01f8-4d59-82f8-a058e84b55f5\") " pod="openshift-ingress-canary/ingress-canary-7v2wh"
May 25 13:06:22.231729 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:22.231468    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:06:22.231729 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:22.231515    2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:06:22.231729 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:22.231532    2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84ccf98695-7zpjl: secret "image-registry-tls" not found
May 25 13:06:22.231729 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:22.231479    2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:06:22.231729 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:22.231583    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls podName:d0b340d4-ffa3-4bf8-aeda-9d13cf88d293 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:26.231560569 +0000 UTC m=+162.234302313 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls") pod "dns-default-27ppb" (UID: "d0b340d4-ffa3-4bf8-aeda-9d13cf88d293") : secret "dns-default-metrics-tls" not found
May 25 13:06:22.231729 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:22.231601    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls podName:4b30c873-1845-484c-ae06-b3eab38d772d nodeName:}" failed. No retries permitted until 2026-05-25 13:07:26.231592679 +0000 UTC m=+162.234334411 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls") pod "image-registry-84ccf98695-7zpjl" (UID: "4b30c873-1845-484c-ae06-b3eab38d772d") : secret "image-registry-tls" not found
May 25 13:06:22.231729 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:22.231614    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert podName:8b9092df-01f8-4d59-82f8-a058e84b55f5 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:26.231606725 +0000 UTC m=+162.234348459 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert") pod "ingress-canary-7v2wh" (UID: "8b9092df-01f8-4d59-82f8-a058e84b55f5") : secret "canary-serving-cert" not found
May 25 13:06:25.808608 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:25.808580    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5ppwf"
May 25 13:06:35.554725 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.554686    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-dzlqc"]
May 25 13:06:35.557497 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.557480    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-dzlqc"
May 25 13:06:35.560030 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.560011    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-l65m6\""
May 25 13:06:35.560900 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.560883    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
May 25 13:06:35.568096 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.567007    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
May 25 13:06:35.568096 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.567296    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-dzlqc"]
May 25 13:06:35.618888 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.618846    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk658\" (UniqueName: \"kubernetes.io/projected/62b6e938-f09a-4ed8-8e4a-a2135505f9a8-kube-api-access-pk658\") pod \"volume-data-source-validator-5cf8c89d64-dzlqc\" (UID: \"62b6e938-f09a-4ed8-8e4a-a2135505f9a8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-dzlqc"
May 25 13:06:35.658814 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.658780    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-8wljq"]
May 25 13:06:35.661717 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.661701    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-8wljq"
May 25 13:06:35.664130 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.664105    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
May 25 13:06:35.664130 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.664106    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
May 25 13:06:35.664321 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.664221    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
May 25 13:06:35.664321 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.664240    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
May 25 13:06:35.664321 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.664246    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-4wzmp\""
May 25 13:06:35.665831 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.665809    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-pkfft"]
May 25 13:06:35.669741 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.669723    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pkfft"
May 25 13:06:35.679915 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.679897    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
May 25 13:06:35.680035 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.679897    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
May 25 13:06:35.680137 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.680120    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
May 25 13:06:35.680792 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.680772    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-8wljq"]
May 25 13:06:35.681466 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.681451    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
May 25 13:06:35.681959 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.681938    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-pkfft"]
May 25 13:06:35.684804 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.684786    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-89bm8\""
May 25 13:06:35.719337 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.719306    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b747p\" (UniqueName: \"kubernetes.io/projected/feba83ac-e3f6-44e8-bd44-14aa150789a2-kube-api-access-b747p\") pod \"console-operator-76dff48d59-8wljq\" (UID: \"feba83ac-e3f6-44e8-bd44-14aa150789a2\") " pod="openshift-console-operator/console-operator-76dff48d59-8wljq"
May 25 13:06:35.719509 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.719429    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/feba83ac-e3f6-44e8-bd44-14aa150789a2-config\") pod \"console-operator-76dff48d59-8wljq\" (UID: \"feba83ac-e3f6-44e8-bd44-14aa150789a2\") " pod="openshift-console-operator/console-operator-76dff48d59-8wljq"
May 25 13:06:35.719509 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.719456    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pk658\" (UniqueName: \"kubernetes.io/projected/62b6e938-f09a-4ed8-8e4a-a2135505f9a8-kube-api-access-pk658\") pod \"volume-data-source-validator-5cf8c89d64-dzlqc\" (UID: \"62b6e938-f09a-4ed8-8e4a-a2135505f9a8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-dzlqc"
May 25 13:06:35.719593 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.719511    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/feba83ac-e3f6-44e8-bd44-14aa150789a2-serving-cert\") pod \"console-operator-76dff48d59-8wljq\" (UID: \"feba83ac-e3f6-44e8-bd44-14aa150789a2\") " pod="openshift-console-operator/console-operator-76dff48d59-8wljq"
May 25 13:06:35.719593 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.719534    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/feba83ac-e3f6-44e8-bd44-14aa150789a2-trusted-ca\") pod \"console-operator-76dff48d59-8wljq\" (UID: \"feba83ac-e3f6-44e8-bd44-14aa150789a2\") " pod="openshift-console-operator/console-operator-76dff48d59-8wljq"
May 25 13:06:35.737545 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.737509    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk658\" (UniqueName: \"kubernetes.io/projected/62b6e938-f09a-4ed8-8e4a-a2135505f9a8-kube-api-access-pk658\") pod \"volume-data-source-validator-5cf8c89d64-dzlqc\" (UID: \"62b6e938-f09a-4ed8-8e4a-a2135505f9a8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-dzlqc"
May 25 13:06:35.773433 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.773395    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7spmp"]
May 25 13:06:35.776392 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.776371    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7spmp"
May 25 13:06:35.776517 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.776440    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq"]
May 25 13:06:35.778829 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.778807    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
May 25 13:06:35.779192 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.779178    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq"
May 25 13:06:35.779357 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.779336    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
May 25 13:06:35.779950 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.779935    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
May 25 13:06:35.780003 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.779943    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-tkplt\""
May 25 13:06:35.780044 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.780006    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
May 25 13:06:35.782677 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.782659    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 25 13:06:35.782777 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.782679    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
May 25 13:06:35.782777 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.782768    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gb879\""
May 25 13:06:35.783223 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.783203    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 25 13:06:35.784355 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.784335    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
May 25 13:06:35.792212 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.791989    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7spmp"]
May 25 13:06:35.792880 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.792859    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq"]
May 25 13:06:35.820750 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.820664    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmz95\" (UniqueName: \"kubernetes.io/projected/44a31a63-dfff-45b5-8a7d-4eed3412ddfa-kube-api-access-nmz95\") pod \"cluster-samples-operator-697587599-pkfft\" (UID: \"44a31a63-dfff-45b5-8a7d-4eed3412ddfa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pkfft"
May 25 13:06:35.820750 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.820710    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/feba83ac-e3f6-44e8-bd44-14aa150789a2-config\") pod \"console-operator-76dff48d59-8wljq\" (UID: \"feba83ac-e3f6-44e8-bd44-14aa150789a2\") " pod="openshift-console-operator/console-operator-76dff48d59-8wljq"
May 25 13:06:35.820750 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.820753    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/44a31a63-dfff-45b5-8a7d-4eed3412ddfa-samples-operator-tls\") pod \"cluster-samples-operator-697587599-pkfft\" (UID: \"44a31a63-dfff-45b5-8a7d-4eed3412ddfa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pkfft"
May 25 13:06:35.821032 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.820774    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/feba83ac-e3f6-44e8-bd44-14aa150789a2-serving-cert\") pod \"console-operator-76dff48d59-8wljq\" (UID: \"feba83ac-e3f6-44e8-bd44-14aa150789a2\") " pod="openshift-console-operator/console-operator-76dff48d59-8wljq"
May 25 13:06:35.821032 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.820794    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/feba83ac-e3f6-44e8-bd44-14aa150789a2-trusted-ca\") pod \"console-operator-76dff48d59-8wljq\" (UID: \"feba83ac-e3f6-44e8-bd44-14aa150789a2\") " pod="openshift-console-operator/console-operator-76dff48d59-8wljq"
May 25 13:06:35.821032 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.820818    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b747p\" (UniqueName: \"kubernetes.io/projected/feba83ac-e3f6-44e8-bd44-14aa150789a2-kube-api-access-b747p\") pod \"console-operator-76dff48d59-8wljq\" (UID: \"feba83ac-e3f6-44e8-bd44-14aa150789a2\") " pod="openshift-console-operator/console-operator-76dff48d59-8wljq"
May 25 13:06:35.821594 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.821570    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/feba83ac-e3f6-44e8-bd44-14aa150789a2-config\") pod \"console-operator-76dff48d59-8wljq\" (UID: \"feba83ac-e3f6-44e8-bd44-14aa150789a2\") " pod="openshift-console-operator/console-operator-76dff48d59-8wljq"
May 25 13:06:35.822078 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.822047    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/feba83ac-e3f6-44e8-bd44-14aa150789a2-trusted-ca\") pod \"console-operator-76dff48d59-8wljq\" (UID: \"feba83ac-e3f6-44e8-bd44-14aa150789a2\") " pod="openshift-console-operator/console-operator-76dff48d59-8wljq"
May 25 13:06:35.823139 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.823118    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/feba83ac-e3f6-44e8-bd44-14aa150789a2-serving-cert\") pod \"console-operator-76dff48d59-8wljq\" (UID: \"feba83ac-e3f6-44e8-bd44-14aa150789a2\") " pod="openshift-console-operator/console-operator-76dff48d59-8wljq"
May 25 13:06:35.833670 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.833642    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b747p\" (UniqueName: \"kubernetes.io/projected/feba83ac-e3f6-44e8-bd44-14aa150789a2-kube-api-access-b747p\") pod \"console-operator-76dff48d59-8wljq\" (UID: \"feba83ac-e3f6-44e8-bd44-14aa150789a2\") " pod="openshift-console-operator/console-operator-76dff48d59-8wljq"
May 25 13:06:35.870583 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.870551    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-dzlqc"
May 25 13:06:35.921896 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.921864    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmw8m\" (UniqueName: \"kubernetes.io/projected/0d406e4f-9b85-403f-b250-242f6d50a43f-kube-api-access-cmw8m\") pod \"kube-storage-version-migrator-operator-565644f49-7spmp\" (UID: \"0d406e4f-9b85-403f-b250-242f6d50a43f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7spmp"
May 25 13:06:35.922056 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.921921    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/44a31a63-dfff-45b5-8a7d-4eed3412ddfa-samples-operator-tls\") pod \"cluster-samples-operator-697587599-pkfft\" (UID: \"44a31a63-dfff-45b5-8a7d-4eed3412ddfa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pkfft"
May 25 13:06:35.922056 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.921978    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-hb7qq\" (UID: \"3d09042c-c990-4d33-8e6c-9aba7f5a94d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq"
May 25 13:06:35.922056 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.922011    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4ghv\" (UniqueName: \"kubernetes.io/projected/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-kube-api-access-g4ghv\") pod \"cluster-monitoring-operator-6c75959455-hb7qq\" (UID: \"3d09042c-c990-4d33-8e6c-9aba7f5a94d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq"
May 25 13:06:35.922056 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:35.922015    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 13:06:35.922251 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:35.922074    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a31a63-dfff-45b5-8a7d-4eed3412ddfa-samples-operator-tls podName:44a31a63-dfff-45b5-8a7d-4eed3412ddfa nodeName:}" failed. No retries permitted until 2026-05-25 13:06:36.422056814 +0000 UTC m=+112.424798546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/44a31a63-dfff-45b5-8a7d-4eed3412ddfa-samples-operator-tls") pod "cluster-samples-operator-697587599-pkfft" (UID: "44a31a63-dfff-45b5-8a7d-4eed3412ddfa") : secret "samples-operator-tls" not found
May 25 13:06:35.922251 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.922095    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0d406e4f-9b85-403f-b250-242f6d50a43f-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-7spmp\" (UID: \"0d406e4f-9b85-403f-b250-242f6d50a43f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7spmp"
May 25 13:06:35.922251 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.922132    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nmz95\" (UniqueName: \"kubernetes.io/projected/44a31a63-dfff-45b5-8a7d-4eed3412ddfa-kube-api-access-nmz95\") pod \"cluster-samples-operator-697587599-pkfft\" (UID: \"44a31a63-dfff-45b5-8a7d-4eed3412ddfa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pkfft"
May 25 13:06:35.922251 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.922187    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0d406e4f-9b85-403f-b250-242f6d50a43f-config\") pod \"kube-storage-version-migrator-operator-565644f49-7spmp\" (UID: \"0d406e4f-9b85-403f-b250-242f6d50a43f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7spmp"
May 25 13:06:35.922251 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.922222    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-hb7qq\" (UID: \"3d09042c-c990-4d33-8e6c-9aba7f5a94d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq"
May 25 13:06:35.932084 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.932002    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmz95\" (UniqueName: \"kubernetes.io/projected/44a31a63-dfff-45b5-8a7d-4eed3412ddfa-kube-api-access-nmz95\") pod \"cluster-samples-operator-697587599-pkfft\" (UID: \"44a31a63-dfff-45b5-8a7d-4eed3412ddfa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pkfft"
May 25 13:06:35.970822 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.970791    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-8wljq"
May 25 13:06:35.993084 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:35.993055    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-dzlqc"]
May 25 13:06:35.996970 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:06:35.996939    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b6e938_f09a_4ed8_8e4a_a2135505f9a8.slice/crio-0c837e3fc3d11262a1afc3baf123f13be81db83b70f92d6e7a6a26fd7cd14a89 WatchSource:0}: Error finding container 0c837e3fc3d11262a1afc3baf123f13be81db83b70f92d6e7a6a26fd7cd14a89: Status 404 returned error can't find the container with id 0c837e3fc3d11262a1afc3baf123f13be81db83b70f92d6e7a6a26fd7cd14a89
May 25 13:06:36.023386 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:36.023296    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-hb7qq\" (UID: \"3d09042c-c990-4d33-8e6c-9aba7f5a94d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq"
May 25 13:06:36.023386 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:36.023336    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g4ghv\" (UniqueName: \"kubernetes.io/projected/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-kube-api-access-g4ghv\") pod \"cluster-monitoring-operator-6c75959455-hb7qq\" (UID: \"3d09042c-c990-4d33-8e6c-9aba7f5a94d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq"
May 25 13:06:36.023386 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:36.023376    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0d406e4f-9b85-403f-b250-242f6d50a43f-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-7spmp\" (UID: \"0d406e4f-9b85-403f-b250-242f6d50a43f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7spmp"
May 25 13:06:36.023703 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:36.023415    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0d406e4f-9b85-403f-b250-242f6d50a43f-config\") pod \"kube-storage-version-migrator-operator-565644f49-7spmp\" (UID: \"0d406e4f-9b85-403f-b250-242f6d50a43f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7spmp"
May 25 13:06:36.023703 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:36.023445    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-hb7qq\" (UID: \"3d09042c-c990-4d33-8e6c-9aba7f5a94d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq"
May 25 13:06:36.023703 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:36.023477    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 13:06:36.023703 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:36.023497    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cmw8m\" (UniqueName: \"kubernetes.io/projected/0d406e4f-9b85-403f-b250-242f6d50a43f-kube-api-access-cmw8m\") pod \"kube-storage-version-migrator-operator-565644f49-7spmp\" (UID: \"0d406e4f-9b85-403f-b250-242f6d50a43f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7spmp"
May 25 13:06:36.023703 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:36.023558    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-cluster-monitoring-operator-tls podName:3d09042c-c990-4d33-8e6c-9aba7f5a94d7 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:36.523535189 +0000 UTC m=+112.526276942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-hb7qq" (UID: "3d09042c-c990-4d33-8e6c-9aba7f5a94d7") : secret "cluster-monitoring-operator-tls" not found
May 25 13:06:36.024555 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:36.024528    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0d406e4f-9b85-403f-b250-242f6d50a43f-config\") pod \"kube-storage-version-migrator-operator-565644f49-7spmp\" (UID: \"0d406e4f-9b85-403f-b250-242f6d50a43f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7spmp"
May 25 13:06:36.024681 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:36.024661    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-hb7qq\" (UID: \"3d09042c-c990-4d33-8e6c-9aba7f5a94d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq"
May 25 13:06:36.026164 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:36.026134    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0d406e4f-9b85-403f-b250-242f6d50a43f-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-7spmp\" (UID: \"0d406e4f-9b85-403f-b250-242f6d50a43f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7spmp"
May 25 13:06:36.032884 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:36.032853    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4ghv\" (UniqueName: \"kubernetes.io/projected/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-kube-api-access-g4ghv\") pod \"cluster-monitoring-operator-6c75959455-hb7qq\" (UID: \"3d09042c-c990-4d33-8e6c-9aba7f5a94d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq"
May 25 13:06:36.033750 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:36.033729    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmw8m\" (UniqueName: \"kubernetes.io/projected/0d406e4f-9b85-403f-b250-242f6d50a43f-kube-api-access-cmw8m\") pod \"kube-storage-version-migrator-operator-565644f49-7spmp\" (UID: \"0d406e4f-9b85-403f-b250-242f6d50a43f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7spmp"
May 25 13:06:36.086692 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:36.086609    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7spmp"
May 25 13:06:36.094490 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:36.094455    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-8wljq"]
May 25 13:06:36.097235 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:06:36.097203    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeba83ac_e3f6_44e8_bd44_14aa150789a2.slice/crio-277247eb8e8f8fccc91ed7bce27455adc70c4ae236fa06453c64f55d0fa94aa4 WatchSource:0}: Error finding container 277247eb8e8f8fccc91ed7bce27455adc70c4ae236fa06453c64f55d0fa94aa4: Status 404 returned error can't find the container with id 277247eb8e8f8fccc91ed7bce27455adc70c4ae236fa06453c64f55d0fa94aa4
May 25 13:06:36.211890 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:36.210993    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7spmp"]
May 25 13:06:36.215645 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:06:36.215605    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d406e4f_9b85_403f_b250_242f6d50a43f.slice/crio-c599c8c1b44d70f9af6cff5551f4e17800d8f6abe9667d3525298c624404683f WatchSource:0}: Error finding container c599c8c1b44d70f9af6cff5551f4e17800d8f6abe9667d3525298c624404683f: Status 404 returned error can't find the container with id c599c8c1b44d70f9af6cff5551f4e17800d8f6abe9667d3525298c624404683f
May 25 13:06:36.427227 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:36.427193    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/44a31a63-dfff-45b5-8a7d-4eed3412ddfa-samples-operator-tls\") pod \"cluster-samples-operator-697587599-pkfft\" (UID: \"44a31a63-dfff-45b5-8a7d-4eed3412ddfa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pkfft"
May 25 13:06:36.427407 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:36.427322    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 13:06:36.427407 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:36.427383    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a31a63-dfff-45b5-8a7d-4eed3412ddfa-samples-operator-tls podName:44a31a63-dfff-45b5-8a7d-4eed3412ddfa nodeName:}" failed. No retries permitted until 2026-05-25 13:06:37.427364451 +0000 UTC m=+113.430106205 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/44a31a63-dfff-45b5-8a7d-4eed3412ddfa-samples-operator-tls") pod "cluster-samples-operator-697587599-pkfft" (UID: "44a31a63-dfff-45b5-8a7d-4eed3412ddfa") : secret "samples-operator-tls" not found
May 25 13:06:36.528589 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:36.528550    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-hb7qq\" (UID: \"3d09042c-c990-4d33-8e6c-9aba7f5a94d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq"
May 25 13:06:36.528770 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:36.528705    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 13:06:36.528770 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:36.528769    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-cluster-monitoring-operator-tls podName:3d09042c-c990-4d33-8e6c-9aba7f5a94d7 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:37.528753838 +0000 UTC m=+113.531495576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-hb7qq" (UID: "3d09042c-c990-4d33-8e6c-9aba7f5a94d7") : secret "cluster-monitoring-operator-tls" not found
May 25 13:06:36.887353 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:36.887285    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-dzlqc" event={"ID":"62b6e938-f09a-4ed8-8e4a-a2135505f9a8","Type":"ContainerStarted","Data":"0c837e3fc3d11262a1afc3baf123f13be81db83b70f92d6e7a6a26fd7cd14a89"}
May 25 13:06:36.889199 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:36.889113    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7spmp" event={"ID":"0d406e4f-9b85-403f-b250-242f6d50a43f","Type":"ContainerStarted","Data":"c599c8c1b44d70f9af6cff5551f4e17800d8f6abe9667d3525298c624404683f"}
May 25 13:06:36.890515 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:36.890482    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-8wljq" event={"ID":"feba83ac-e3f6-44e8-bd44-14aa150789a2","Type":"ContainerStarted","Data":"277247eb8e8f8fccc91ed7bce27455adc70c4ae236fa06453c64f55d0fa94aa4"}
May 25 13:06:37.436908 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:37.436871    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/44a31a63-dfff-45b5-8a7d-4eed3412ddfa-samples-operator-tls\") pod \"cluster-samples-operator-697587599-pkfft\" (UID: \"44a31a63-dfff-45b5-8a7d-4eed3412ddfa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pkfft"
May 25 13:06:37.437076 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:37.437047    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 13:06:37.437139 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:37.437125    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a31a63-dfff-45b5-8a7d-4eed3412ddfa-samples-operator-tls podName:44a31a63-dfff-45b5-8a7d-4eed3412ddfa nodeName:}" failed. No retries permitted until 2026-05-25 13:06:39.437104099 +0000 UTC m=+115.439845834 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/44a31a63-dfff-45b5-8a7d-4eed3412ddfa-samples-operator-tls") pod "cluster-samples-operator-697587599-pkfft" (UID: "44a31a63-dfff-45b5-8a7d-4eed3412ddfa") : secret "samples-operator-tls" not found
May 25 13:06:37.537928 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:37.537881    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-hb7qq\" (UID: \"3d09042c-c990-4d33-8e6c-9aba7f5a94d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq"
May 25 13:06:37.538110 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:37.537987    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 13:06:37.538110 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:37.538061    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-cluster-monitoring-operator-tls podName:3d09042c-c990-4d33-8e6c-9aba7f5a94d7 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:39.538043405 +0000 UTC m=+115.540785157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-hb7qq" (UID: "3d09042c-c990-4d33-8e6c-9aba7f5a94d7") : secret "cluster-monitoring-operator-tls" not found
May 25 13:06:37.894031 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:37.893994    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-dzlqc" event={"ID":"62b6e938-f09a-4ed8-8e4a-a2135505f9a8","Type":"ContainerStarted","Data":"90e8000cd90f4e00bac16a1ff228082b3cf4ef54ac5cd5b4c381af6d7f1de5b7"}
May 25 13:06:37.911047 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:37.910990    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-dzlqc" podStartSLOduration=1.577889217 podStartE2EDuration="2.910975511s" podCreationTimestamp="2026-05-25 13:06:35 +0000 UTC" firstStartedPulling="2026-05-25 13:06:35.998712682 +0000 UTC m=+112.001454414" lastFinishedPulling="2026-05-25 13:06:37.33179897 +0000 UTC m=+113.334540708" observedRunningTime="2026-05-25 13:06:37.910715831 +0000 UTC m=+113.913457583" watchObservedRunningTime="2026-05-25 13:06:37.910975511 +0000 UTC m=+113.913717265"
May 25 13:06:38.897949 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:38.897912    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7spmp" event={"ID":"0d406e4f-9b85-403f-b250-242f6d50a43f","Type":"ContainerStarted","Data":"7c71d7c85714d972d7e5f9bf0681997268f6c7e3f4306b94904bfd503bc58f04"}
May 25 13:06:38.899350 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:38.899327    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/0.log"
May 25 13:06:38.899479 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:38.899367    2569 generic.go:358] "Generic (PLEG): container finished" podID="feba83ac-e3f6-44e8-bd44-14aa150789a2" containerID="2d36255ad8cd79eaba8e09640ba37a0bc772713676e2baa58afff0dbd86757d3" exitCode=255
May 25 13:06:38.899479 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:38.899440    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-8wljq" event={"ID":"feba83ac-e3f6-44e8-bd44-14aa150789a2","Type":"ContainerDied","Data":"2d36255ad8cd79eaba8e09640ba37a0bc772713676e2baa58afff0dbd86757d3"}
May 25 13:06:38.899610 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:38.899593    2569 scope.go:117] "RemoveContainer" containerID="2d36255ad8cd79eaba8e09640ba37a0bc772713676e2baa58afff0dbd86757d3"
May 25 13:06:38.920322 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:38.920268    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7spmp" podStartSLOduration=1.773336762 podStartE2EDuration="3.920249844s" podCreationTimestamp="2026-05-25 13:06:35 +0000 UTC" firstStartedPulling="2026-05-25 13:06:36.217362386 +0000 UTC m=+112.220104118" lastFinishedPulling="2026-05-25 13:06:38.364275469 +0000 UTC m=+114.367017200" observedRunningTime="2026-05-25 13:06:38.918985907 +0000 UTC m=+114.921727673" watchObservedRunningTime="2026-05-25 13:06:38.920249844 +0000 UTC m=+114.922991601"
May 25 13:06:39.455157 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:39.455125    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/44a31a63-dfff-45b5-8a7d-4eed3412ddfa-samples-operator-tls\") pod \"cluster-samples-operator-697587599-pkfft\" (UID: \"44a31a63-dfff-45b5-8a7d-4eed3412ddfa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pkfft"
May 25 13:06:39.455332 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:39.455279    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 13:06:39.455380 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:39.455345    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a31a63-dfff-45b5-8a7d-4eed3412ddfa-samples-operator-tls podName:44a31a63-dfff-45b5-8a7d-4eed3412ddfa nodeName:}" failed. No retries permitted until 2026-05-25 13:06:43.455328488 +0000 UTC m=+119.458070221 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/44a31a63-dfff-45b5-8a7d-4eed3412ddfa-samples-operator-tls") pod "cluster-samples-operator-697587599-pkfft" (UID: "44a31a63-dfff-45b5-8a7d-4eed3412ddfa") : secret "samples-operator-tls" not found
May 25 13:06:39.556529 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:39.556487    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-hb7qq\" (UID: \"3d09042c-c990-4d33-8e6c-9aba7f5a94d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq"
May 25 13:06:39.556702 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:39.556604    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 13:06:39.556702 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:39.556658    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-cluster-monitoring-operator-tls podName:3d09042c-c990-4d33-8e6c-9aba7f5a94d7 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:43.556644321 +0000 UTC m=+119.559386065 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-hb7qq" (UID: "3d09042c-c990-4d33-8e6c-9aba7f5a94d7") : secret "cluster-monitoring-operator-tls" not found
May 25 13:06:39.586870 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:39.586838    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-j448b"]
May 25 13:06:39.592149 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:39.592120    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-j448b"
May 25 13:06:39.595704 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:39.595683    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-s7zkj\""
May 25 13:06:39.596578 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:39.596559    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
May 25 13:06:39.597215 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:39.597195    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
May 25 13:06:39.604309 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:39.604286    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-j448b"]
May 25 13:06:39.657279 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:39.657241    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f74j2\" (UniqueName: \"kubernetes.io/projected/0ad976e4-ddc2-4b60-b40f-d20b5b8a6b96-kube-api-access-f74j2\") pod \"migrator-b985f5d89-j448b\" (UID: \"0ad976e4-ddc2-4b60-b40f-d20b5b8a6b96\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-j448b"
May 25 13:06:39.757691 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:39.757609    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f74j2\" (UniqueName: \"kubernetes.io/projected/0ad976e4-ddc2-4b60-b40f-d20b5b8a6b96-kube-api-access-f74j2\") pod \"migrator-b985f5d89-j448b\" (UID: \"0ad976e4-ddc2-4b60-b40f-d20b5b8a6b96\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-j448b"
May 25 13:06:39.765947 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:39.765913    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f74j2\" (UniqueName: \"kubernetes.io/projected/0ad976e4-ddc2-4b60-b40f-d20b5b8a6b96-kube-api-access-f74j2\") pod \"migrator-b985f5d89-j448b\" (UID: \"0ad976e4-ddc2-4b60-b40f-d20b5b8a6b96\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-j448b"
May 25 13:06:39.900505 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:39.900470    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-j448b"
May 25 13:06:39.903458 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:39.903436    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/1.log"
May 25 13:06:39.903823 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:39.903807    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/0.log"
May 25 13:06:39.903878 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:39.903842    2569 generic.go:358] "Generic (PLEG): container finished" podID="feba83ac-e3f6-44e8-bd44-14aa150789a2" containerID="e225c4f787700fc382f0928b67c503cac89c8b5f68310e4c1da998aff9dfa903" exitCode=255
May 25 13:06:39.903917 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:39.903875    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-8wljq" event={"ID":"feba83ac-e3f6-44e8-bd44-14aa150789a2","Type":"ContainerDied","Data":"e225c4f787700fc382f0928b67c503cac89c8b5f68310e4c1da998aff9dfa903"}
May 25 13:06:39.903959 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:39.903916    2569 scope.go:117] "RemoveContainer" containerID="2d36255ad8cd79eaba8e09640ba37a0bc772713676e2baa58afff0dbd86757d3"
May 25 13:06:39.904398 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:39.904378    2569 scope.go:117] "RemoveContainer" containerID="e225c4f787700fc382f0928b67c503cac89c8b5f68310e4c1da998aff9dfa903"
May 25 13:06:39.904607 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:39.904580    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-8wljq_openshift-console-operator(feba83ac-e3f6-44e8-bd44-14aa150789a2)\"" pod="openshift-console-operator/console-operator-76dff48d59-8wljq" podUID="feba83ac-e3f6-44e8-bd44-14aa150789a2"
May 25 13:06:40.020705 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:40.020626    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-j448b"]
May 25 13:06:40.023770 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:06:40.023738    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad976e4_ddc2_4b60_b40f_d20b5b8a6b96.slice/crio-b3053806f120a272e6619afe256c689b097f7dc1c15a54cc52743d304957764b WatchSource:0}: Error finding container b3053806f120a272e6619afe256c689b097f7dc1c15a54cc52743d304957764b: Status 404 returned error can't find the container with id b3053806f120a272e6619afe256c689b097f7dc1c15a54cc52743d304957764b
May 25 13:06:40.908529 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:40.908498    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/1.log"
May 25 13:06:40.909069 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:40.908922    2569 scope.go:117] "RemoveContainer" containerID="e225c4f787700fc382f0928b67c503cac89c8b5f68310e4c1da998aff9dfa903"
May 25 13:06:40.909160 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:40.909123    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-8wljq_openshift-console-operator(feba83ac-e3f6-44e8-bd44-14aa150789a2)\"" pod="openshift-console-operator/console-operator-76dff48d59-8wljq" podUID="feba83ac-e3f6-44e8-bd44-14aa150789a2"
May 25 13:06:40.909909 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:40.909883    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-j448b" event={"ID":"0ad976e4-ddc2-4b60-b40f-d20b5b8a6b96","Type":"ContainerStarted","Data":"b3053806f120a272e6619afe256c689b097f7dc1c15a54cc52743d304957764b"}
May 25 13:06:41.916141 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:41.916109    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-j448b" event={"ID":"0ad976e4-ddc2-4b60-b40f-d20b5b8a6b96","Type":"ContainerStarted","Data":"175fb63e4dadedd8c999e3bb94d66ee9d8e79c11d8e236d93bb507913aeadff5"}
May 25 13:06:41.916141 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:41.916146    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-j448b" event={"ID":"0ad976e4-ddc2-4b60-b40f-d20b5b8a6b96","Type":"ContainerStarted","Data":"645ca947c0740b1b2dcc3bd6e2db672f3cdc0209623ce820202e10c92d314454"}
May 25 13:06:41.933518 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:41.933463    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-j448b" podStartSLOduration=1.897777635 podStartE2EDuration="2.933447866s" podCreationTimestamp="2026-05-25 13:06:39 +0000 UTC" firstStartedPulling="2026-05-25 13:06:40.025553735 +0000 UTC m=+116.028295471" lastFinishedPulling="2026-05-25 13:06:41.061223966 +0000 UTC m=+117.063965702" observedRunningTime="2026-05-25 13:06:41.932604282 +0000 UTC m=+117.935346037" watchObservedRunningTime="2026-05-25 13:06:41.933447866 +0000 UTC m=+117.936189620"
May 25 13:06:42.426845 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:42.426817    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pz7qk_217de8ce-783f-48b4-bee9-82bbe61f2450/dns-node-resolver/0.log"
May 25 13:06:42.506164 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:42.506127    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-lkzsx"]
May 25 13:06:42.509076 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:42.509060    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-lkzsx"
May 25 13:06:42.511574 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:42.511552    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-qtt2j\""
May 25 13:06:42.511574 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:42.511569    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
May 25 13:06:42.512730 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:42.512703    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
May 25 13:06:42.512730 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:42.512719    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
May 25 13:06:42.512913 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:42.512751    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
May 25 13:06:42.517935 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:42.517906    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-lkzsx"]
May 25 13:06:42.585205 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:42.585149    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4041bf52-92da-4ce7-affb-598d3ed8b7cd-signing-key\") pod \"service-ca-67d667c7d-lkzsx\" (UID: \"4041bf52-92da-4ce7-affb-598d3ed8b7cd\") " pod="openshift-service-ca/service-ca-67d667c7d-lkzsx"
May 25 13:06:42.585385 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:42.585274    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8h57\" (UniqueName: \"kubernetes.io/projected/4041bf52-92da-4ce7-affb-598d3ed8b7cd-kube-api-access-d8h57\") pod \"service-ca-67d667c7d-lkzsx\" (UID: \"4041bf52-92da-4ce7-affb-598d3ed8b7cd\") " pod="openshift-service-ca/service-ca-67d667c7d-lkzsx"
May 25 13:06:42.585385 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:42.585339    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4041bf52-92da-4ce7-affb-598d3ed8b7cd-signing-cabundle\") pod \"service-ca-67d667c7d-lkzsx\" (UID: \"4041bf52-92da-4ce7-affb-598d3ed8b7cd\") " pod="openshift-service-ca/service-ca-67d667c7d-lkzsx"
May 25 13:06:42.686049 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:42.685958    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4041bf52-92da-4ce7-affb-598d3ed8b7cd-signing-cabundle\") pod \"service-ca-67d667c7d-lkzsx\" (UID: \"4041bf52-92da-4ce7-affb-598d3ed8b7cd\") " pod="openshift-service-ca/service-ca-67d667c7d-lkzsx"
May 25 13:06:42.686049 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:42.686027    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4041bf52-92da-4ce7-affb-598d3ed8b7cd-signing-key\") pod \"service-ca-67d667c7d-lkzsx\" (UID: \"4041bf52-92da-4ce7-affb-598d3ed8b7cd\") " pod="openshift-service-ca/service-ca-67d667c7d-lkzsx"
May 25 13:06:42.686301 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:42.686220    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d8h57\" (UniqueName: \"kubernetes.io/projected/4041bf52-92da-4ce7-affb-598d3ed8b7cd-kube-api-access-d8h57\") pod \"service-ca-67d667c7d-lkzsx\" (UID: \"4041bf52-92da-4ce7-affb-598d3ed8b7cd\") " pod="openshift-service-ca/service-ca-67d667c7d-lkzsx"
May 25 13:06:42.686716 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:42.686698    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4041bf52-92da-4ce7-affb-598d3ed8b7cd-signing-cabundle\") pod \"service-ca-67d667c7d-lkzsx\" (UID: \"4041bf52-92da-4ce7-affb-598d3ed8b7cd\") " pod="openshift-service-ca/service-ca-67d667c7d-lkzsx"
May 25 13:06:42.688344 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:42.688328    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4041bf52-92da-4ce7-affb-598d3ed8b7cd-signing-key\") pod \"service-ca-67d667c7d-lkzsx\" (UID: \"4041bf52-92da-4ce7-affb-598d3ed8b7cd\") " pod="openshift-service-ca/service-ca-67d667c7d-lkzsx"
May 25 13:06:42.700304 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:42.700282    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8h57\" (UniqueName: \"kubernetes.io/projected/4041bf52-92da-4ce7-affb-598d3ed8b7cd-kube-api-access-d8h57\") pod \"service-ca-67d667c7d-lkzsx\" (UID: \"4041bf52-92da-4ce7-affb-598d3ed8b7cd\") " pod="openshift-service-ca/service-ca-67d667c7d-lkzsx"
May 25 13:06:42.818586 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:42.818541    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-lkzsx"
May 25 13:06:42.943477 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:42.943396    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-lkzsx"]
May 25 13:06:42.947042 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:06:42.947006    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4041bf52_92da_4ce7_affb_598d3ed8b7cd.slice/crio-5b128a90ec896c9d723563e9b9d5d307441c4d93fb98843472eb625fc6aa8892 WatchSource:0}: Error finding container 5b128a90ec896c9d723563e9b9d5d307441c4d93fb98843472eb625fc6aa8892: Status 404 returned error can't find the container with id 5b128a90ec896c9d723563e9b9d5d307441c4d93fb98843472eb625fc6aa8892
May 25 13:06:43.212716 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:43.212642    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c2xxn_d4c54f53-586b-496f-bd70-663617ee0a12/node-ca/0.log"
May 25 13:06:43.494528 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:43.494440    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/44a31a63-dfff-45b5-8a7d-4eed3412ddfa-samples-operator-tls\") pod \"cluster-samples-operator-697587599-pkfft\" (UID: \"44a31a63-dfff-45b5-8a7d-4eed3412ddfa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pkfft"
May 25 13:06:43.494692 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:43.494598    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 13:06:43.494692 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:43.494680    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a31a63-dfff-45b5-8a7d-4eed3412ddfa-samples-operator-tls podName:44a31a63-dfff-45b5-8a7d-4eed3412ddfa nodeName:}" failed. No retries permitted until 2026-05-25 13:06:51.494658192 +0000 UTC m=+127.497399927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/44a31a63-dfff-45b5-8a7d-4eed3412ddfa-samples-operator-tls") pod "cluster-samples-operator-697587599-pkfft" (UID: "44a31a63-dfff-45b5-8a7d-4eed3412ddfa") : secret "samples-operator-tls" not found
May 25 13:06:43.595095 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:43.595063    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-hb7qq\" (UID: \"3d09042c-c990-4d33-8e6c-9aba7f5a94d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq"
May 25 13:06:43.595295 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:43.595264    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 13:06:43.595358 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:43.595347    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-cluster-monitoring-operator-tls podName:3d09042c-c990-4d33-8e6c-9aba7f5a94d7 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:51.595325859 +0000 UTC m=+127.598067609 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-hb7qq" (UID: "3d09042c-c990-4d33-8e6c-9aba7f5a94d7") : secret "cluster-monitoring-operator-tls" not found
May 25 13:06:43.922515 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:43.922475    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-lkzsx" event={"ID":"4041bf52-92da-4ce7-affb-598d3ed8b7cd","Type":"ContainerStarted","Data":"5b128a90ec896c9d723563e9b9d5d307441c4d93fb98843472eb625fc6aa8892"}
May 25 13:06:44.611817 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:44.611787    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-j448b_0ad976e4-ddc2-4b60-b40f-d20b5b8a6b96/migrator/0.log"
May 25 13:06:44.808320 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:44.808253    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-j448b_0ad976e4-ddc2-4b60-b40f-d20b5b8a6b96/graceful-termination/0.log"
May 25 13:06:44.925921 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:44.925885    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-lkzsx" event={"ID":"4041bf52-92da-4ce7-affb-598d3ed8b7cd","Type":"ContainerStarted","Data":"4ca37735b9a2c6f61cb976cd65f6ba834cc4e02603db53e784702f2ee31ba25c"}
May 25 13:06:44.944948 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:44.944885    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-67d667c7d-lkzsx" podStartSLOduration=1.437015035 podStartE2EDuration="2.944869375s" podCreationTimestamp="2026-05-25 13:06:42 +0000 UTC" firstStartedPulling="2026-05-25 13:06:42.94890617 +0000 UTC m=+118.951647901" lastFinishedPulling="2026-05-25 13:06:44.456760506 +0000 UTC m=+120.459502241" observedRunningTime="2026-05-25 13:06:44.944288855 +0000 UTC m=+120.947030609" watchObservedRunningTime="2026-05-25 13:06:44.944869375 +0000 UTC m=+120.947611129"
May 25 13:06:45.012536 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:45.012506    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-7spmp_0d406e4f-9b85-403f-b250-242f6d50a43f/kube-storage-version-migrator-operator/0.log"
May 25 13:06:45.970917 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:45.970879    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-8wljq"
May 25 13:06:45.970917 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:45.970927    2569 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-8wljq"
May 25 13:06:45.971497 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:45.971396    2569 scope.go:117] "RemoveContainer" containerID="e225c4f787700fc382f0928b67c503cac89c8b5f68310e4c1da998aff9dfa903"
May 25 13:06:45.971632 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:45.971609    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-8wljq_openshift-console-operator(feba83ac-e3f6-44e8-bd44-14aa150789a2)\"" pod="openshift-console-operator/console-operator-76dff48d59-8wljq" podUID="feba83ac-e3f6-44e8-bd44-14aa150789a2"
May 25 13:06:51.566977 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:51.566931    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/44a31a63-dfff-45b5-8a7d-4eed3412ddfa-samples-operator-tls\") pod \"cluster-samples-operator-697587599-pkfft\" (UID: \"44a31a63-dfff-45b5-8a7d-4eed3412ddfa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pkfft"
May 25 13:06:51.569320 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:51.569295    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/44a31a63-dfff-45b5-8a7d-4eed3412ddfa-samples-operator-tls\") pod \"cluster-samples-operator-697587599-pkfft\" (UID: \"44a31a63-dfff-45b5-8a7d-4eed3412ddfa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pkfft"
May 25 13:06:51.579751 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:51.579721    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-89bm8\""
May 25 13:06:51.587753 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:51.587732    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pkfft"
May 25 13:06:51.668472 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:51.668069    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-hb7qq\" (UID: \"3d09042c-c990-4d33-8e6c-9aba7f5a94d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq"
May 25 13:06:51.668796 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:51.668583    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 13:06:51.668796 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:51.668660    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-cluster-monitoring-operator-tls podName:3d09042c-c990-4d33-8e6c-9aba7f5a94d7 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:07.668637719 +0000 UTC m=+143.671379466 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-hb7qq" (UID: "3d09042c-c990-4d33-8e6c-9aba7f5a94d7") : secret "cluster-monitoring-operator-tls" not found
May 25 13:06:51.716310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:51.716275    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-pkfft"]
May 25 13:06:51.943661 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:51.943620    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pkfft" event={"ID":"44a31a63-dfff-45b5-8a7d-4eed3412ddfa","Type":"ContainerStarted","Data":"aa5b28e593b19a80d4ff5eeaf9d83e3b8b17700060545e2057305dea9d4305ec"}
May 25 13:06:53.949998 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:53.949960    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pkfft" event={"ID":"44a31a63-dfff-45b5-8a7d-4eed3412ddfa","Type":"ContainerStarted","Data":"3f3c88efe74f6e8babd234bd4888e23669651352273602412122960414d7cfa8"}
May 25 13:06:53.949998 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:53.949997    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pkfft" event={"ID":"44a31a63-dfff-45b5-8a7d-4eed3412ddfa","Type":"ContainerStarted","Data":"467fdf82cf216ab0a99b77c4aa734ef8ae2e2d69a42573e509cc9e9c8974640d"}
May 25 13:06:53.968737 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:53.968679    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pkfft" podStartSLOduration=17.402943057 podStartE2EDuration="18.96866024s" podCreationTimestamp="2026-05-25 13:06:35 +0000 UTC" firstStartedPulling="2026-05-25 13:06:51.755919653 +0000 UTC m=+127.758661390" lastFinishedPulling="2026-05-25 13:06:53.321636842 +0000 UTC m=+129.324378573" observedRunningTime="2026-05-25 13:06:53.967372179 +0000 UTC m=+129.970113932" watchObservedRunningTime="2026-05-25 13:06:53.96866024 +0000 UTC m=+129.971401996"
May 25 13:06:54.491221 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:54.491187    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs\") pod \"network-metrics-daemon-8rxxc\" (UID: \"8878a1b9-7848-4f10-a0f6-f88059209e87\") " pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:06:54.491385 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:54.491333    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 25 13:06:54.491427 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:06:54.491398    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs podName:8878a1b9-7848-4f10-a0f6-f88059209e87 nodeName:}" failed. No retries permitted until 2026-05-25 13:08:56.491382755 +0000 UTC m=+252.494124486 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs") pod "network-metrics-daemon-8rxxc" (UID: "8878a1b9-7848-4f10-a0f6-f88059209e87") : secret "metrics-daemon-secret" not found
May 25 13:06:57.592403 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:57.592370    2569 scope.go:117] "RemoveContainer" containerID="e225c4f787700fc382f0928b67c503cac89c8b5f68310e4c1da998aff9dfa903"
May 25 13:06:57.961942 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:57.961901    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/1.log"
May 25 13:06:57.962115 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:57.961990    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-8wljq" event={"ID":"feba83ac-e3f6-44e8-bd44-14aa150789a2","Type":"ContainerStarted","Data":"da6c928837c91627e961481243549a1d61fa29a4410c17539e06a638a9dd4a12"}
May 25 13:06:57.962419 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:57.962393    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-8wljq"
May 25 13:06:57.981736 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:57.981683    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-76dff48d59-8wljq" podStartSLOduration=20.719652124 podStartE2EDuration="22.981670074s" podCreationTimestamp="2026-05-25 13:06:35 +0000 UTC" firstStartedPulling="2026-05-25 13:06:36.099066064 +0000 UTC m=+112.101807797" lastFinishedPulling="2026-05-25 13:06:38.36108401 +0000 UTC m=+114.363825747" observedRunningTime="2026-05-25 13:06:57.980114816 +0000 UTC m=+133.982856569" watchObservedRunningTime="2026-05-25 13:06:57.981670074 +0000 UTC m=+133.984411877"
May 25 13:06:58.274939 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:06:58.274679    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-76dff48d59-8wljq"
May 25 13:07:02.873409 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:02.873367    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5qn7d"]
May 25 13:07:02.876616 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:02.876592    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5qn7d"
May 25 13:07:02.882015 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:02.881990    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 25 13:07:02.882137 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:02.882005    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 25 13:07:02.882137 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:02.882081    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bzxd9\""
May 25 13:07:02.882137 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:02.882038    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 25 13:07:02.882360 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:02.882006    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 25 13:07:02.909636 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:02.909599    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5qn7d"]
May 25 13:07:02.960371 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:02.960337    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce26be33-f950-446a-9800-9b471d4c8bfe-data-volume\") pod \"insights-runtime-extractor-5qn7d\" (UID: \"ce26be33-f950-446a-9800-9b471d4c8bfe\") " pod="openshift-insights/insights-runtime-extractor-5qn7d"
May 25 13:07:02.960537 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:02.960396    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce26be33-f950-446a-9800-9b471d4c8bfe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qn7d\" (UID: \"ce26be33-f950-446a-9800-9b471d4c8bfe\") " pod="openshift-insights/insights-runtime-extractor-5qn7d"
May 25 13:07:02.960537 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:02.960418    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ce26be33-f950-446a-9800-9b471d4c8bfe-crio-socket\") pod \"insights-runtime-extractor-5qn7d\" (UID: \"ce26be33-f950-446a-9800-9b471d4c8bfe\") " pod="openshift-insights/insights-runtime-extractor-5qn7d"
May 25 13:07:02.960537 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:02.960485    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svprs\" (UniqueName: \"kubernetes.io/projected/ce26be33-f950-446a-9800-9b471d4c8bfe-kube-api-access-svprs\") pod \"insights-runtime-extractor-5qn7d\" (UID: \"ce26be33-f950-446a-9800-9b471d4c8bfe\") " pod="openshift-insights/insights-runtime-extractor-5qn7d"
May 25 13:07:02.960656 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:02.960614    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce26be33-f950-446a-9800-9b471d4c8bfe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5qn7d\" (UID: \"ce26be33-f950-446a-9800-9b471d4c8bfe\") " pod="openshift-insights/insights-runtime-extractor-5qn7d"
May 25 13:07:02.977014 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:02.976980    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-57bd96b888-gw2fw"]
May 25 13:07:02.979959 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:02.979943    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-gw2fw"
May 25 13:07:02.982888 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:02.982865    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-jjm6h\""
May 25 13:07:02.983040 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:02.983024    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 25 13:07:02.984637 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:02.984622    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 25 13:07:02.993985 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:02.993966    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-gw2fw"]
May 25 13:07:03.061807 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:03.061768    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce26be33-f950-446a-9800-9b471d4c8bfe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5qn7d\" (UID: \"ce26be33-f950-446a-9800-9b471d4c8bfe\") " pod="openshift-insights/insights-runtime-extractor-5qn7d"
May 25 13:07:03.061807 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:03.061813    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce26be33-f950-446a-9800-9b471d4c8bfe-data-volume\") pod \"insights-runtime-extractor-5qn7d\" (UID: \"ce26be33-f950-446a-9800-9b471d4c8bfe\") " pod="openshift-insights/insights-runtime-extractor-5qn7d"
May 25 13:07:03.062080 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:03.061861    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce26be33-f950-446a-9800-9b471d4c8bfe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qn7d\" (UID: \"ce26be33-f950-446a-9800-9b471d4c8bfe\") " pod="openshift-insights/insights-runtime-extractor-5qn7d"
May 25 13:07:03.062080 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:03.061889    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ce26be33-f950-446a-9800-9b471d4c8bfe-crio-socket\") pod \"insights-runtime-extractor-5qn7d\" (UID: \"ce26be33-f950-446a-9800-9b471d4c8bfe\") " pod="openshift-insights/insights-runtime-extractor-5qn7d"
May 25 13:07:03.062080 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:03.061959    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ce26be33-f950-446a-9800-9b471d4c8bfe-crio-socket\") pod \"insights-runtime-extractor-5qn7d\" (UID: \"ce26be33-f950-446a-9800-9b471d4c8bfe\") " pod="openshift-insights/insights-runtime-extractor-5qn7d"
May 25 13:07:03.062080 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:03.061975    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsqc6\" (UniqueName: \"kubernetes.io/projected/b17a540d-e750-4fc7-b014-b5905162bad2-kube-api-access-xsqc6\") pod \"downloads-57bd96b888-gw2fw\" (UID: \"b17a540d-e750-4fc7-b014-b5905162bad2\") " pod="openshift-console/downloads-57bd96b888-gw2fw"
May 25 13:07:03.062080 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:03.062042    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-svprs\" (UniqueName: \"kubernetes.io/projected/ce26be33-f950-446a-9800-9b471d4c8bfe-kube-api-access-svprs\") pod \"insights-runtime-extractor-5qn7d\" (UID: \"ce26be33-f950-446a-9800-9b471d4c8bfe\") " pod="openshift-insights/insights-runtime-extractor-5qn7d"
May 25 13:07:03.062309 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:03.062292    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce26be33-f950-446a-9800-9b471d4c8bfe-data-volume\") pod \"insights-runtime-extractor-5qn7d\" (UID: \"ce26be33-f950-446a-9800-9b471d4c8bfe\") " pod="openshift-insights/insights-runtime-extractor-5qn7d"
May 25 13:07:03.062437 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:03.062423    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce26be33-f950-446a-9800-9b471d4c8bfe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5qn7d\" (UID: \"ce26be33-f950-446a-9800-9b471d4c8bfe\") " pod="openshift-insights/insights-runtime-extractor-5qn7d"
May 25 13:07:03.064288 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:03.064268    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce26be33-f950-446a-9800-9b471d4c8bfe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qn7d\" (UID: \"ce26be33-f950-446a-9800-9b471d4c8bfe\") " pod="openshift-insights/insights-runtime-extractor-5qn7d"
May 25 13:07:03.078080 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:03.078054    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svprs\" (UniqueName: \"kubernetes.io/projected/ce26be33-f950-446a-9800-9b471d4c8bfe-kube-api-access-svprs\") pod \"insights-runtime-extractor-5qn7d\" (UID: \"ce26be33-f950-446a-9800-9b471d4c8bfe\") " pod="openshift-insights/insights-runtime-extractor-5qn7d"
May 25 13:07:03.163347 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:03.163257    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xsqc6\" (UniqueName: \"kubernetes.io/projected/b17a540d-e750-4fc7-b014-b5905162bad2-kube-api-access-xsqc6\") pod \"downloads-57bd96b888-gw2fw\" (UID: \"b17a540d-e750-4fc7-b014-b5905162bad2\") " pod="openshift-console/downloads-57bd96b888-gw2fw"
May 25 13:07:03.175633 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:03.175599    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsqc6\" (UniqueName: \"kubernetes.io/projected/b17a540d-e750-4fc7-b014-b5905162bad2-kube-api-access-xsqc6\") pod \"downloads-57bd96b888-gw2fw\" (UID: \"b17a540d-e750-4fc7-b014-b5905162bad2\") " pod="openshift-console/downloads-57bd96b888-gw2fw"
May 25 13:07:03.186383 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:03.186358    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5qn7d"
May 25 13:07:03.288674 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:03.288542    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-gw2fw"
May 25 13:07:03.333629 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:03.333574    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5qn7d"]
May 25 13:07:03.418308 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:03.418238    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-gw2fw"]
May 25 13:07:03.421818 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:07:03.421764    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb17a540d_e750_4fc7_b014_b5905162bad2.slice/crio-3c577517538665e57b3ef89cfaf743ad9b8e734e2f4d52f078e82d81334ea2c0 WatchSource:0}: Error finding container 3c577517538665e57b3ef89cfaf743ad9b8e734e2f4d52f078e82d81334ea2c0: Status 404 returned error can't find the container with id 3c577517538665e57b3ef89cfaf743ad9b8e734e2f4d52f078e82d81334ea2c0
May 25 13:07:03.978030 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:03.977992    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-gw2fw" event={"ID":"b17a540d-e750-4fc7-b014-b5905162bad2","Type":"ContainerStarted","Data":"3c577517538665e57b3ef89cfaf743ad9b8e734e2f4d52f078e82d81334ea2c0"}
May 25 13:07:03.979425 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:03.979392    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5qn7d" event={"ID":"ce26be33-f950-446a-9800-9b471d4c8bfe","Type":"ContainerStarted","Data":"295a484d17b5bddacc4da6efd8f8382945fd60f293da0ad38a7d38bacac4d1e1"}
May 25 13:07:03.979556 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:03.979431    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5qn7d" event={"ID":"ce26be33-f950-446a-9800-9b471d4c8bfe","Type":"ContainerStarted","Data":"1373216d798a3e7dcc038f2963aa2cf8cc1462311a1c7840f4072dc1150e2097"}
May 25 13:07:04.984804 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:04.984760    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5qn7d" event={"ID":"ce26be33-f950-446a-9800-9b471d4c8bfe","Type":"ContainerStarted","Data":"1837873626a250b813efd5738c9c9832c459d18d685cb3e7aa7e73f5387b4227"}
May 25 13:07:05.989386 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:05.989344    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5qn7d" event={"ID":"ce26be33-f950-446a-9800-9b471d4c8bfe","Type":"ContainerStarted","Data":"332d3f550e78cc98692d74b63d3f486cec3f780d864ebc96335a60d243a8e3f0"}
May 25 13:07:06.009936 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:06.009870    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5qn7d" podStartSLOduration=1.886853484 podStartE2EDuration="4.009855791s" podCreationTimestamp="2026-05-25 13:07:02 +0000 UTC" firstStartedPulling="2026-05-25 13:07:03.398764685 +0000 UTC m=+139.401506419" lastFinishedPulling="2026-05-25 13:07:05.521766983 +0000 UTC m=+141.524508726" observedRunningTime="2026-05-25 13:07:06.009477884 +0000 UTC m=+142.012219696" watchObservedRunningTime="2026-05-25 13:07:06.009855791 +0000 UTC m=+142.012597544"
May 25 13:07:07.702627 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:07.702588    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-hb7qq\" (UID: \"3d09042c-c990-4d33-8e6c-9aba7f5a94d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq"
May 25 13:07:07.705354 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:07.705325    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d09042c-c990-4d33-8e6c-9aba7f5a94d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-hb7qq\" (UID: \"3d09042c-c990-4d33-8e6c-9aba7f5a94d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq"
May 25 13:07:07.893828 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:07.893793    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gb879\""
May 25 13:07:07.901892 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:07.901864    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq"
May 25 13:07:08.044810 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:08.044774    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq"]
May 25 13:07:08.048926 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:07:08.048896    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d09042c_c990_4d33_8e6c_9aba7f5a94d7.slice/crio-747fdc5860398528f43b1103b0a3ad25602befc5845ceff6107cbac2d01122e2 WatchSource:0}: Error finding container 747fdc5860398528f43b1103b0a3ad25602befc5845ceff6107cbac2d01122e2: Status 404 returned error can't find the container with id 747fdc5860398528f43b1103b0a3ad25602befc5845ceff6107cbac2d01122e2
May 25 13:07:09.003208 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:09.003128    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq" event={"ID":"3d09042c-c990-4d33-8e6c-9aba7f5a94d7","Type":"ContainerStarted","Data":"747fdc5860398528f43b1103b0a3ad25602befc5845ceff6107cbac2d01122e2"}
May 25 13:07:10.007577 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:10.007495    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq" event={"ID":"3d09042c-c990-4d33-8e6c-9aba7f5a94d7","Type":"ContainerStarted","Data":"f0bfef3877e340eb2c6e7e9a2eb25637b4762b6ddbbd00901096ddb7b73defbb"}
May 25 13:07:10.029613 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:10.029549    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-hb7qq" podStartSLOduration=33.38336143 podStartE2EDuration="35.029529939s" podCreationTimestamp="2026-05-25 13:06:35 +0000 UTC" firstStartedPulling="2026-05-25 13:07:08.051271412 +0000 UTC m=+144.054013145" lastFinishedPulling="2026-05-25 13:07:09.697439908 +0000 UTC m=+145.700181654" observedRunningTime="2026-05-25 13:07:10.027724379 +0000 UTC m=+146.030466133" watchObservedRunningTime="2026-05-25 13:07:10.029529939 +0000 UTC m=+146.032271695"
May 25 13:07:10.226725 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:10.226689    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-dzgcp"]
May 25 13:07:10.229988 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:10.229962    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-dzgcp"
May 25 13:07:10.232410 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:10.232385    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
May 25 13:07:10.232532 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:10.232387    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lbjq7\""
May 25 13:07:10.238948 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:10.238901    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-dzgcp"]
May 25 13:07:10.326152 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:10.326065    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ad52974f-66ee-4440-b0ca-065f6d759892-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-dzgcp\" (UID: \"ad52974f-66ee-4440-b0ca-065f6d759892\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-dzgcp"
May 25 13:07:10.426883 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:10.426843    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ad52974f-66ee-4440-b0ca-065f6d759892-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-dzgcp\" (UID: \"ad52974f-66ee-4440-b0ca-065f6d759892\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-dzgcp"
May 25 13:07:10.429800 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:10.429770    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ad52974f-66ee-4440-b0ca-065f6d759892-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-dzgcp\" (UID: \"ad52974f-66ee-4440-b0ca-065f6d759892\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-dzgcp"
May 25 13:07:10.541064 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:10.541023    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-dzgcp"
May 25 13:07:10.686317 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:10.686285    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-dzgcp"]
May 25 13:07:10.690187 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:07:10.690130    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad52974f_66ee_4440_b0ca_065f6d759892.slice/crio-a95ee3656fa81a2433659f5cf665d7c76265af5b45573cb3973a0354c234ef0b WatchSource:0}: Error finding container a95ee3656fa81a2433659f5cf665d7c76265af5b45573cb3973a0354c234ef0b: Status 404 returned error can't find the container with id a95ee3656fa81a2433659f5cf665d7c76265af5b45573cb3973a0354c234ef0b
May 25 13:07:11.012254 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:11.012134    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-dzgcp" event={"ID":"ad52974f-66ee-4440-b0ca-065f6d759892","Type":"ContainerStarted","Data":"a95ee3656fa81a2433659f5cf665d7c76265af5b45573cb3973a0354c234ef0b"}
May 25 13:07:12.016355 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:12.016263    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-dzgcp" event={"ID":"ad52974f-66ee-4440-b0ca-065f6d759892","Type":"ContainerStarted","Data":"171b672dcc4033b89419769c8767f5ff27b334e9d33a88efcc859401be471242"}
May 25 13:07:12.016746 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:12.016479    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-dzgcp"
May 25 13:07:12.021996 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:12.021970    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-dzgcp"
May 25 13:07:12.033158 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:12.033097    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-dzgcp" podStartSLOduration=1.011143777 podStartE2EDuration="2.033075359s" podCreationTimestamp="2026-05-25 13:07:10 +0000 UTC" firstStartedPulling="2026-05-25 13:07:10.692485416 +0000 UTC m=+146.695227153" lastFinishedPulling="2026-05-25 13:07:11.714416982 +0000 UTC m=+147.717158735" observedRunningTime="2026-05-25 13:07:12.032577529 +0000 UTC m=+148.035319283" watchObservedRunningTime="2026-05-25 13:07:12.033075359 +0000 UTC m=+148.035817115"
May 25 13:07:13.385954 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.385922    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-8644f8cf75-c69k8"]
May 25 13:07:13.413377 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.413348    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8644f8cf75-c69k8"]
May 25 13:07:13.413538 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.413490    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:13.416005 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.415981    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 25 13:07:13.416986 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.416924    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 25 13:07:13.416986 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.416948    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 25 13:07:13.416986 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.416967    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 25 13:07:13.416986 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.416977    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 25 13:07:13.417292 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.416933    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8jjtw\""
May 25 13:07:13.454922 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.454883    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bff5617f-f53f-4d47-afb9-4b68d3b159ed-console-serving-cert\") pod \"console-8644f8cf75-c69k8\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") " pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:13.455120 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.454932    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bff5617f-f53f-4d47-afb9-4b68d3b159ed-service-ca\") pod \"console-8644f8cf75-c69k8\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") " pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:13.455120 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.454960    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4fvr\" (UniqueName: \"kubernetes.io/projected/bff5617f-f53f-4d47-afb9-4b68d3b159ed-kube-api-access-l4fvr\") pod \"console-8644f8cf75-c69k8\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") " pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:13.455120 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.455108    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bff5617f-f53f-4d47-afb9-4b68d3b159ed-oauth-serving-cert\") pod \"console-8644f8cf75-c69k8\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") " pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:13.455311 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.455145    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bff5617f-f53f-4d47-afb9-4b68d3b159ed-console-config\") pod \"console-8644f8cf75-c69k8\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") " pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:13.455311 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.455245    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bff5617f-f53f-4d47-afb9-4b68d3b159ed-console-oauth-config\") pod \"console-8644f8cf75-c69k8\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") " pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:13.556021 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.555985    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bff5617f-f53f-4d47-afb9-4b68d3b159ed-console-oauth-config\") pod \"console-8644f8cf75-c69k8\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") " pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:13.556021 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.556024    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bff5617f-f53f-4d47-afb9-4b68d3b159ed-console-serving-cert\") pod \"console-8644f8cf75-c69k8\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") " pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:13.556327 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.556045    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bff5617f-f53f-4d47-afb9-4b68d3b159ed-service-ca\") pod \"console-8644f8cf75-c69k8\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") " pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:13.556327 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.556062    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l4fvr\" (UniqueName: \"kubernetes.io/projected/bff5617f-f53f-4d47-afb9-4b68d3b159ed-kube-api-access-l4fvr\") pod \"console-8644f8cf75-c69k8\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") " pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:13.556327 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.556190    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bff5617f-f53f-4d47-afb9-4b68d3b159ed-oauth-serving-cert\") pod \"console-8644f8cf75-c69k8\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") " pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:13.556327 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.556220    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bff5617f-f53f-4d47-afb9-4b68d3b159ed-console-config\") pod \"console-8644f8cf75-c69k8\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") " pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:13.556964 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.556884    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bff5617f-f53f-4d47-afb9-4b68d3b159ed-service-ca\") pod \"console-8644f8cf75-c69k8\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") " pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:13.557078 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.556992    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bff5617f-f53f-4d47-afb9-4b68d3b159ed-oauth-serving-cert\") pod \"console-8644f8cf75-c69k8\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") " pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:13.557078 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.557005    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bff5617f-f53f-4d47-afb9-4b68d3b159ed-console-config\") pod \"console-8644f8cf75-c69k8\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") " pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:13.558746 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.558723    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bff5617f-f53f-4d47-afb9-4b68d3b159ed-console-oauth-config\") pod \"console-8644f8cf75-c69k8\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") " pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:13.559100 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.559075    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bff5617f-f53f-4d47-afb9-4b68d3b159ed-console-serving-cert\") pod \"console-8644f8cf75-c69k8\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") " pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:13.564347 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.564323    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4fvr\" (UniqueName: \"kubernetes.io/projected/bff5617f-f53f-4d47-afb9-4b68d3b159ed-kube-api-access-l4fvr\") pod \"console-8644f8cf75-c69k8\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") " pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:13.724658 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:13.724541    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:16.674299 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.674258    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-8w47r"]
May 25 13:07:16.678939 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.678905    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-8w47r"
May 25 13:07:16.681563 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.681383    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 25 13:07:16.681563 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.681444    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 25 13:07:16.681563 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.681383    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 25 13:07:16.681563 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.681508    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mvsg9\""
May 25 13:07:16.694978 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.694720    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-8w47r"]
May 25 13:07:16.707635 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.707597    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"]
May 25 13:07:16.711481 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.711456    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"
May 25 13:07:16.714108 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.713917    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
May 25 13:07:16.714108 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.714058    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-94lql\""
May 25 13:07:16.714380 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.714337    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
May 25 13:07:16.714449 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.714389    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
May 25 13:07:16.719388 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.718863    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-x7b5z"]
May 25 13:07:16.722393 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.722371    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.724950 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.724918    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 25 13:07:16.725145 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.725127    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 25 13:07:16.725281 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.725138    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 25 13:07:16.726080 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.726061    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"]
May 25 13:07:16.726934 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.726915    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fvln4\""
May 25 13:07:16.788496 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.788441    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1152b71-6f95-4a41-87e0-98fccf6c5512-node-exporter-accelerators-collector-config\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.788496 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.788496    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1152b71-6f95-4a41-87e0-98fccf6c5512-root\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.788725 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.788530    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff44f516-493d-4bab-8b9c-841c79d799a7-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-znb4d\" (UID: \"ff44f516-493d-4bab-8b9c-841c79d799a7\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"
May 25 13:07:16.788725 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.788609    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1152b71-6f95-4a41-87e0-98fccf6c5512-node-exporter-wtmp\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.788725 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.788634    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tjkn\" (UniqueName: \"kubernetes.io/projected/f1152b71-6f95-4a41-87e0-98fccf6c5512-kube-api-access-9tjkn\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.789348 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.789272    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8e9b997-2768-47e6-82fb-5f498706590e-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-8w47r\" (UID: \"b8e9b997-2768-47e6-82fb-5f498706590e\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8w47r"
May 25 13:07:16.789463 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.789350    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw68t\" (UniqueName: \"kubernetes.io/projected/b8e9b997-2768-47e6-82fb-5f498706590e-kube-api-access-bw68t\") pod \"openshift-state-metrics-9c959559f-8w47r\" (UID: \"b8e9b997-2768-47e6-82fb-5f498706590e\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8w47r"
May 25 13:07:16.789463 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.789393    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1152b71-6f95-4a41-87e0-98fccf6c5512-node-exporter-textfile\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.789463 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.789445    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8e9b997-2768-47e6-82fb-5f498706590e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-8w47r\" (UID: \"b8e9b997-2768-47e6-82fb-5f498706590e\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8w47r"
May 25 13:07:16.789463 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.789463    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1152b71-6f95-4a41-87e0-98fccf6c5512-node-exporter-tls\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.789805 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.789483    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1152b71-6f95-4a41-87e0-98fccf6c5512-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.789805 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.789529    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm4kx\" (UniqueName: \"kubernetes.io/projected/ff44f516-493d-4bab-8b9c-841c79d799a7-kube-api-access-gm4kx\") pod \"kube-state-metrics-54fd6bc5c4-znb4d\" (UID: \"ff44f516-493d-4bab-8b9c-841c79d799a7\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"
May 25 13:07:16.789805 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.789560    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ff44f516-493d-4bab-8b9c-841c79d799a7-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-znb4d\" (UID: \"ff44f516-493d-4bab-8b9c-841c79d799a7\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"
May 25 13:07:16.789805 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.789591    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1152b71-6f95-4a41-87e0-98fccf6c5512-sys\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.789805 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.789633    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1152b71-6f95-4a41-87e0-98fccf6c5512-metrics-client-ca\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.789805 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.789736    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8e9b997-2768-47e6-82fb-5f498706590e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-8w47r\" (UID: \"b8e9b997-2768-47e6-82fb-5f498706590e\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8w47r"
May 25 13:07:16.789805 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.789759    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff44f516-493d-4bab-8b9c-841c79d799a7-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-znb4d\" (UID: \"ff44f516-493d-4bab-8b9c-841c79d799a7\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"
May 25 13:07:16.789805 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.789776    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff44f516-493d-4bab-8b9c-841c79d799a7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-znb4d\" (UID: \"ff44f516-493d-4bab-8b9c-841c79d799a7\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"
May 25 13:07:16.790197 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.789830    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ff44f516-493d-4bab-8b9c-841c79d799a7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-znb4d\" (UID: \"ff44f516-493d-4bab-8b9c-841c79d799a7\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"
May 25 13:07:16.890879 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.890839    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8e9b997-2768-47e6-82fb-5f498706590e-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-8w47r\" (UID: \"b8e9b997-2768-47e6-82fb-5f498706590e\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8w47r"
May 25 13:07:16.891046 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.890889    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bw68t\" (UniqueName: \"kubernetes.io/projected/b8e9b997-2768-47e6-82fb-5f498706590e-kube-api-access-bw68t\") pod \"openshift-state-metrics-9c959559f-8w47r\" (UID: \"b8e9b997-2768-47e6-82fb-5f498706590e\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8w47r"
May 25 13:07:16.891046 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.890929    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1152b71-6f95-4a41-87e0-98fccf6c5512-node-exporter-textfile\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.891046 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.890979    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8e9b997-2768-47e6-82fb-5f498706590e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-8w47r\" (UID: \"b8e9b997-2768-47e6-82fb-5f498706590e\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8w47r"
May 25 13:07:16.891046 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.891006    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1152b71-6f95-4a41-87e0-98fccf6c5512-node-exporter-tls\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.891046 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.891033    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1152b71-6f95-4a41-87e0-98fccf6c5512-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.891336 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.891065    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gm4kx\" (UniqueName: \"kubernetes.io/projected/ff44f516-493d-4bab-8b9c-841c79d799a7-kube-api-access-gm4kx\") pod \"kube-state-metrics-54fd6bc5c4-znb4d\" (UID: \"ff44f516-493d-4bab-8b9c-841c79d799a7\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"
May 25 13:07:16.891336 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.891095    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ff44f516-493d-4bab-8b9c-841c79d799a7-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-znb4d\" (UID: \"ff44f516-493d-4bab-8b9c-841c79d799a7\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"
May 25 13:07:16.891336 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.891122    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1152b71-6f95-4a41-87e0-98fccf6c5512-sys\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.891336 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.891149    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1152b71-6f95-4a41-87e0-98fccf6c5512-metrics-client-ca\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.891336 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.891202    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8e9b997-2768-47e6-82fb-5f498706590e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-8w47r\" (UID: \"b8e9b997-2768-47e6-82fb-5f498706590e\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8w47r"
May 25 13:07:16.891336 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.891230    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff44f516-493d-4bab-8b9c-841c79d799a7-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-znb4d\" (UID: \"ff44f516-493d-4bab-8b9c-841c79d799a7\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"
May 25 13:07:16.891336 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.891256    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff44f516-493d-4bab-8b9c-841c79d799a7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-znb4d\" (UID: \"ff44f516-493d-4bab-8b9c-841c79d799a7\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"
May 25 13:07:16.891336 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.891318    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ff44f516-493d-4bab-8b9c-841c79d799a7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-znb4d\" (UID: \"ff44f516-493d-4bab-8b9c-841c79d799a7\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"
May 25 13:07:16.891720 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.891364    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1152b71-6f95-4a41-87e0-98fccf6c5512-sys\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.891720 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.891366    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1152b71-6f95-4a41-87e0-98fccf6c5512-node-exporter-accelerators-collector-config\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.891720 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.891426    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1152b71-6f95-4a41-87e0-98fccf6c5512-root\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.891720 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.891459    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff44f516-493d-4bab-8b9c-841c79d799a7-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-znb4d\" (UID: \"ff44f516-493d-4bab-8b9c-841c79d799a7\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"
May 25 13:07:16.891720 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.891505    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1152b71-6f95-4a41-87e0-98fccf6c5512-node-exporter-wtmp\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.891720 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.891532    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9tjkn\" (UniqueName: \"kubernetes.io/projected/f1152b71-6f95-4a41-87e0-98fccf6c5512-kube-api-access-9tjkn\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.892028 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.891882    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1152b71-6f95-4a41-87e0-98fccf6c5512-node-exporter-accelerators-collector-config\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.893021 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.892263    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ff44f516-493d-4bab-8b9c-841c79d799a7-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-znb4d\" (UID: \"ff44f516-493d-4bab-8b9c-841c79d799a7\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"
May 25 13:07:16.893021 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.892298    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff44f516-493d-4bab-8b9c-841c79d799a7-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-znb4d\" (UID: \"ff44f516-493d-4bab-8b9c-841c79d799a7\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"
May 25 13:07:16.893021 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.892324    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1152b71-6f95-4a41-87e0-98fccf6c5512-root\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.893021 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:07:16.892393    2569 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
May 25 13:07:16.893021 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.892424    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1152b71-6f95-4a41-87e0-98fccf6c5512-node-exporter-textfile\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.893021 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:07:16.892455    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff44f516-493d-4bab-8b9c-841c79d799a7-kube-state-metrics-tls podName:ff44f516-493d-4bab-8b9c-841c79d799a7 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:17.392437051 +0000 UTC m=+153.395178786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ff44f516-493d-4bab-8b9c-841c79d799a7-kube-state-metrics-tls") pod "kube-state-metrics-54fd6bc5c4-znb4d" (UID: "ff44f516-493d-4bab-8b9c-841c79d799a7") : secret "kube-state-metrics-tls" not found
May 25 13:07:16.893021 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.891508    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8e9b997-2768-47e6-82fb-5f498706590e-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-8w47r\" (UID: \"b8e9b997-2768-47e6-82fb-5f498706590e\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8w47r"
May 25 13:07:16.893021 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:07:16.892516    2569 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
May 25 13:07:16.893021 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:07:16.892547    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8e9b997-2768-47e6-82fb-5f498706590e-openshift-state-metrics-tls podName:b8e9b997-2768-47e6-82fb-5f498706590e nodeName:}" failed. No retries permitted until 2026-05-25 13:07:17.392536983 +0000 UTC m=+153.395278715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b8e9b997-2768-47e6-82fb-5f498706590e-openshift-state-metrics-tls") pod "openshift-state-metrics-9c959559f-8w47r" (UID: "b8e9b997-2768-47e6-82fb-5f498706590e") : secret "openshift-state-metrics-tls" not found
May 25 13:07:16.893021 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.892582    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1152b71-6f95-4a41-87e0-98fccf6c5512-node-exporter-wtmp\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.893021 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.892768    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1152b71-6f95-4a41-87e0-98fccf6c5512-metrics-client-ca\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.893021 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:07:16.892870    2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 25 13:07:16.893021 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:07:16.892922    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1152b71-6f95-4a41-87e0-98fccf6c5512-node-exporter-tls podName:f1152b71-6f95-4a41-87e0-98fccf6c5512 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:17.392907798 +0000 UTC m=+153.395649538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f1152b71-6f95-4a41-87e0-98fccf6c5512-node-exporter-tls") pod "node-exporter-x7b5z" (UID: "f1152b71-6f95-4a41-87e0-98fccf6c5512") : secret "node-exporter-tls" not found
May 25 13:07:16.893739 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.893265    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ff44f516-493d-4bab-8b9c-841c79d799a7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-znb4d\" (UID: \"ff44f516-493d-4bab-8b9c-841c79d799a7\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"
May 25 13:07:16.894086 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.894067    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1152b71-6f95-4a41-87e0-98fccf6c5512-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:16.894237 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.894163    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8e9b997-2768-47e6-82fb-5f498706590e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-8w47r\" (UID: \"b8e9b997-2768-47e6-82fb-5f498706590e\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8w47r"
May 25 13:07:16.894556 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.894535    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff44f516-493d-4bab-8b9c-841c79d799a7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-znb4d\" (UID: \"ff44f516-493d-4bab-8b9c-841c79d799a7\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"
May 25 13:07:16.906309 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.906264    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw68t\" (UniqueName: \"kubernetes.io/projected/b8e9b997-2768-47e6-82fb-5f498706590e-kube-api-access-bw68t\") pod \"openshift-state-metrics-9c959559f-8w47r\" (UID: \"b8e9b997-2768-47e6-82fb-5f498706590e\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8w47r"
May 25 13:07:16.916230 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.915103    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm4kx\" (UniqueName: \"kubernetes.io/projected/ff44f516-493d-4bab-8b9c-841c79d799a7-kube-api-access-gm4kx\") pod \"kube-state-metrics-54fd6bc5c4-znb4d\" (UID: \"ff44f516-493d-4bab-8b9c-841c79d799a7\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"
May 25 13:07:16.921067 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:16.921005    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tjkn\" (UniqueName: \"kubernetes.io/projected/f1152b71-6f95-4a41-87e0-98fccf6c5512-kube-api-access-9tjkn\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:17.397548 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:17.397510    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8e9b997-2768-47e6-82fb-5f498706590e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-8w47r\" (UID: \"b8e9b997-2768-47e6-82fb-5f498706590e\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8w47r"
May 25 13:07:17.397732 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:17.397621    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff44f516-493d-4bab-8b9c-841c79d799a7-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-znb4d\" (UID: \"ff44f516-493d-4bab-8b9c-841c79d799a7\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"
May 25 13:07:17.397732 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:07:17.397663    2569 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
May 25 13:07:17.397732 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:17.397698    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1152b71-6f95-4a41-87e0-98fccf6c5512-node-exporter-tls\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:17.397879 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:07:17.397749    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8e9b997-2768-47e6-82fb-5f498706590e-openshift-state-metrics-tls podName:b8e9b997-2768-47e6-82fb-5f498706590e nodeName:}" failed. No retries permitted until 2026-05-25 13:07:18.397713594 +0000 UTC m=+154.400455326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b8e9b997-2768-47e6-82fb-5f498706590e-openshift-state-metrics-tls") pod "openshift-state-metrics-9c959559f-8w47r" (UID: "b8e9b997-2768-47e6-82fb-5f498706590e") : secret "openshift-state-metrics-tls" not found
May 25 13:07:17.400420 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:17.400392    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1152b71-6f95-4a41-87e0-98fccf6c5512-node-exporter-tls\") pod \"node-exporter-x7b5z\" (UID: \"f1152b71-6f95-4a41-87e0-98fccf6c5512\") " pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:17.400666 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:17.400643    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff44f516-493d-4bab-8b9c-841c79d799a7-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-znb4d\" (UID: \"ff44f516-493d-4bab-8b9c-841c79d799a7\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"
May 25 13:07:17.625064 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:17.625025    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"
May 25 13:07:17.635972 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:17.635944    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x7b5z"
May 25 13:07:18.407925 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:18.407876    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8e9b997-2768-47e6-82fb-5f498706590e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-8w47r\" (UID: \"b8e9b997-2768-47e6-82fb-5f498706590e\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8w47r"
May 25 13:07:18.410685 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:18.410657    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8e9b997-2768-47e6-82fb-5f498706590e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-8w47r\" (UID: \"b8e9b997-2768-47e6-82fb-5f498706590e\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8w47r"
May 25 13:07:18.501993 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:18.501955    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-8w47r"
May 25 13:07:19.726392 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.726364    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-75d748b86c-j78qn"]
May 25 13:07:19.733415 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.733384    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.735996 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.735776    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
May 25 13:07:19.735996 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.735782    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
May 25 13:07:19.736226 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.736022    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
May 25 13:07:19.736226 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.736127    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4e9859ss1asjg\""
May 25 13:07:19.736333 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.736267    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
May 25 13:07:19.736380 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.736359    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
May 25 13:07:19.736468 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.736446    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-bc9xp\""
May 25 13:07:19.740633 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.740596    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75d748b86c-j78qn"]
May 25 13:07:19.821900 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.821848    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5109856c-fa52-4fc8-855d-ca1dff292b04-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.822075 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.821916    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5109856c-fa52-4fc8-855d-ca1dff292b04-secret-grpc-tls\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.822075 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.821964    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5109856c-fa52-4fc8-855d-ca1dff292b04-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.822075 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.821991    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzj7w\" (UniqueName: \"kubernetes.io/projected/5109856c-fa52-4fc8-855d-ca1dff292b04-kube-api-access-pzj7w\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.822075 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.822064    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5109856c-fa52-4fc8-855d-ca1dff292b04-secret-thanos-querier-tls\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.822325 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.822186    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5109856c-fa52-4fc8-855d-ca1dff292b04-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.822325 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.822239    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5109856c-fa52-4fc8-855d-ca1dff292b04-metrics-client-ca\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.822325 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.822278    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5109856c-fa52-4fc8-855d-ca1dff292b04-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.924980 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.923072    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5109856c-fa52-4fc8-855d-ca1dff292b04-secret-thanos-querier-tls\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.924980 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.923148    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5109856c-fa52-4fc8-855d-ca1dff292b04-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.924980 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.923213    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5109856c-fa52-4fc8-855d-ca1dff292b04-metrics-client-ca\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.924980 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.923264    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5109856c-fa52-4fc8-855d-ca1dff292b04-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.924980 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.923368    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5109856c-fa52-4fc8-855d-ca1dff292b04-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.924980 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.923424    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5109856c-fa52-4fc8-855d-ca1dff292b04-secret-grpc-tls\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.924980 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.923478    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5109856c-fa52-4fc8-855d-ca1dff292b04-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.924980 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.923515    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pzj7w\" (UniqueName: \"kubernetes.io/projected/5109856c-fa52-4fc8-855d-ca1dff292b04-kube-api-access-pzj7w\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.926688 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.926347    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5109856c-fa52-4fc8-855d-ca1dff292b04-metrics-client-ca\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.938668 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.938606    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5109856c-fa52-4fc8-855d-ca1dff292b04-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.940773 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.940199    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5109856c-fa52-4fc8-855d-ca1dff292b04-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.940773 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.940566    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5109856c-fa52-4fc8-855d-ca1dff292b04-secret-grpc-tls\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.944506 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.944451    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzj7w\" (UniqueName: \"kubernetes.io/projected/5109856c-fa52-4fc8-855d-ca1dff292b04-kube-api-access-pzj7w\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.944711 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.944508    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5109856c-fa52-4fc8-855d-ca1dff292b04-secret-thanos-querier-tls\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.945755 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.945731    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5109856c-fa52-4fc8-855d-ca1dff292b04-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:19.946669 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:19.946647    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5109856c-fa52-4fc8-855d-ca1dff292b04-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75d748b86c-j78qn\" (UID: \"5109856c-fa52-4fc8-855d-ca1dff292b04\") " pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:20.040003 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:20.039316    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x7b5z" event={"ID":"f1152b71-6f95-4a41-87e0-98fccf6c5512","Type":"ContainerStarted","Data":"afeef990964e69cef257380a8131bd58a8248f816862e578b9e83d751fd0fb70"}
May 25 13:07:20.046758 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:20.046115    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:20.075249 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:20.075190    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8644f8cf75-c69k8"]
May 25 13:07:20.077601 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:07:20.077562    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff5617f_f53f_4d47_afb9_4b68d3b159ed.slice/crio-4c51b3ad605670c893eb791dd94fcc037454b96a91a0b5460684a1937c235b2e WatchSource:0}: Error finding container 4c51b3ad605670c893eb791dd94fcc037454b96a91a0b5460684a1937c235b2e: Status 404 returned error can't find the container with id 4c51b3ad605670c893eb791dd94fcc037454b96a91a0b5460684a1937c235b2e
May 25 13:07:20.186939 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:20.186901    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75d748b86c-j78qn"]
May 25 13:07:20.190127 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:07:20.190093    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5109856c_fa52_4fc8_855d_ca1dff292b04.slice/crio-14228a17f24ba5d3a8c05a6efead220f0369f4546005aaa3746eb8ad4aef29ef WatchSource:0}: Error finding container 14228a17f24ba5d3a8c05a6efead220f0369f4546005aaa3746eb8ad4aef29ef: Status 404 returned error can't find the container with id 14228a17f24ba5d3a8c05a6efead220f0369f4546005aaa3746eb8ad4aef29ef
May 25 13:07:20.303503 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:20.303476    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d"]
May 25 13:07:20.306099 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:07:20.306069    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff44f516_493d_4bab_8b9c_841c79d799a7.slice/crio-159f8e517f3560da9f7bb09764547717be82a1511f1fe0e8d673d8dedda45740 WatchSource:0}: Error finding container 159f8e517f3560da9f7bb09764547717be82a1511f1fe0e8d673d8dedda45740: Status 404 returned error can't find the container with id 159f8e517f3560da9f7bb09764547717be82a1511f1fe0e8d673d8dedda45740
May 25 13:07:20.308755 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:20.308689    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-8w47r"]
May 25 13:07:20.311436 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:07:20.311399    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e9b997_2768_47e6_82fb_5f498706590e.slice/crio-9fdb59fd6cc64f74e48590a648da03190b10511ba620c5e86c862114f30f4014 WatchSource:0}: Error finding container 9fdb59fd6cc64f74e48590a648da03190b10511ba620c5e86c862114f30f4014: Status 404 returned error can't find the container with id 9fdb59fd6cc64f74e48590a648da03190b10511ba620c5e86c862114f30f4014
May 25 13:07:21.049692 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:21.049613    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn" event={"ID":"5109856c-fa52-4fc8-855d-ca1dff292b04","Type":"ContainerStarted","Data":"14228a17f24ba5d3a8c05a6efead220f0369f4546005aaa3746eb8ad4aef29ef"}
May 25 13:07:21.051144 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:21.051018    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d" event={"ID":"ff44f516-493d-4bab-8b9c-841c79d799a7","Type":"ContainerStarted","Data":"159f8e517f3560da9f7bb09764547717be82a1511f1fe0e8d673d8dedda45740"}
May 25 13:07:21.055973 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:21.052930    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-gw2fw" event={"ID":"b17a540d-e750-4fc7-b014-b5905162bad2","Type":"ContainerStarted","Data":"ab5458a9123a8a0db0ebea62dbabb0a8c3ccd6b0c0516affa90c439601c3d7a9"}
May 25 13:07:21.055973 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:21.055765    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-57bd96b888-gw2fw"
May 25 13:07:21.057788 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:21.057741    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8644f8cf75-c69k8" event={"ID":"bff5617f-f53f-4d47-afb9-4b68d3b159ed","Type":"ContainerStarted","Data":"4c51b3ad605670c893eb791dd94fcc037454b96a91a0b5460684a1937c235b2e"}
May 25 13:07:21.066863 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:21.065126    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-8w47r" event={"ID":"b8e9b997-2768-47e6-82fb-5f498706590e","Type":"ContainerStarted","Data":"ecd6fd01505fa24966e8d5b5d586a9eb453bdb93cf0642118f8643d57308b5b0"}
May 25 13:07:21.068072 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:21.067702    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-8w47r" event={"ID":"b8e9b997-2768-47e6-82fb-5f498706590e","Type":"ContainerStarted","Data":"aa45558482af05e9fba4ca6267a409123a87c37fbdf951ca7b55bcc5a876726a"}
May 25 13:07:21.068072 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:21.067732    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-8w47r" event={"ID":"b8e9b997-2768-47e6-82fb-5f498706590e","Type":"ContainerStarted","Data":"9fdb59fd6cc64f74e48590a648da03190b10511ba620c5e86c862114f30f4014"}
May 25 13:07:21.069372 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:21.069339    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-57bd96b888-gw2fw"
May 25 13:07:21.100717 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:21.098580    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-57bd96b888-gw2fw" podStartSLOduration=2.482978059 podStartE2EDuration="19.098562447s" podCreationTimestamp="2026-05-25 13:07:02 +0000 UTC" firstStartedPulling="2026-05-25 13:07:03.423844995 +0000 UTC m=+139.426586727" lastFinishedPulling="2026-05-25 13:07:20.039429381 +0000 UTC m=+156.042171115" observedRunningTime="2026-05-25 13:07:21.073715879 +0000 UTC m=+157.076457633" watchObservedRunningTime="2026-05-25 13:07:21.098562447 +0000 UTC m=+157.101304204"
May 25 13:07:21.327184 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:07:21.327028    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-84ccf98695-7zpjl" podUID="4b30c873-1845-484c-ae06-b3eab38d772d"
May 25 13:07:21.339408 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:07:21.339333    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-27ppb" podUID="d0b340d4-ffa3-4bf8-aeda-9d13cf88d293"
May 25 13:07:21.345433 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:07:21.345363    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-7v2wh" podUID="8b9092df-01f8-4d59-82f8-a058e84b55f5"
May 25 13:07:21.605210 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:07:21.605125    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-8rxxc" podUID="8878a1b9-7848-4f10-a0f6-f88059209e87"
May 25 13:07:22.076745 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.076683    2569 generic.go:358] "Generic (PLEG): container finished" podID="f1152b71-6f95-4a41-87e0-98fccf6c5512" containerID="80f5e1ba7ccae6de60efd6d82127d18aea1bc5b9f7ea889ee1121a4ceaff25d5" exitCode=0
May 25 13:07:22.077300 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.076872    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x7b5z" event={"ID":"f1152b71-6f95-4a41-87e0-98fccf6c5512","Type":"ContainerDied","Data":"80f5e1ba7ccae6de60efd6d82127d18aea1bc5b9f7ea889ee1121a4ceaff25d5"}
May 25 13:07:22.077300 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.076967    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7v2wh"
May 25 13:07:22.077300 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.077286    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-27ppb"
May 25 13:07:22.077482 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.077444    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:07:22.911927 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.911885    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 13:07:22.938697 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.938655    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 13:07:22.939520 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.939481    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:22.944326 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.944282    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 25 13:07:22.944500 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.944343    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 25 13:07:22.944662 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.944291    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 25 13:07:22.944662 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.944630    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 25 13:07:22.944820 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.944730    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 25 13:07:22.944975 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.944829    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 25 13:07:22.945062 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.945020    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 25 13:07:22.945062 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.945056    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hv275\""
May 25 13:07:22.945310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.945295    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 25 13:07:22.945414 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.945340    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 25 13:07:22.945576 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.945558    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 25 13:07:22.945576 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.945572    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 25 13:07:22.945867 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.945819    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-947586grgijv7\""
May 25 13:07:22.951635 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.951263    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 25 13:07:22.954777 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:22.954755    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 25 13:07:23.009645 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.009605    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8644f8cf75-c69k8"]
May 25 13:07:23.066195 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.066140    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.066503 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.066480    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.066798 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.066661    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-config\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.066798 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.066695    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.066798 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.066734    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.066798 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.066764    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.067015 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.066911    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.067015 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.066979    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.067114 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.067041    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.067114 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.067065    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vlzb\" (UniqueName: \"kubernetes.io/projected/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-kube-api-access-8vlzb\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.067114 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.067100    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.067266 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.067122    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-web-config\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.067266 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.067156    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-config-out\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.067266 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.067193    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.067266 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.067215    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.067266 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.067247    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.067501 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.067274    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.067501 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.067298    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.168291 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.168199    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.168291 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.168268    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.168994 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.168297    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.168994 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.168326    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.168994 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.168381    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.168994 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.168919    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8vlzb\" (UniqueName: \"kubernetes.io/projected/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-kube-api-access-8vlzb\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.168994 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.168985    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.169280 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.169020    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-web-config\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.169280 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.169057    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-config-out\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.169280 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.169081    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.169280 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.169111    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.169280 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.169116    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.169280 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.169211    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.169280 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.169243    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.169280 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.169275    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.169835 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.169326    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.169835 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.169418    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.169835 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.169457    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.169835 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.169473    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-config\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.169835 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.169521    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.170903 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.170561    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.170903 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.170579    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.171249 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.171224    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.181460 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.175133    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.181460 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.175584    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-config\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.181460 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.176065    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.181460 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.177156    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.181460 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.177543    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.181460 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.177596    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.181460 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.178093    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.181460 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.178670    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.181460 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.178796    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.181460 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.179999    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-config-out\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.181460 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.180587    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.181460 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.181412    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-web-config\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.188590 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.188524    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vlzb\" (UniqueName: \"kubernetes.io/projected/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-kube-api-access-8vlzb\") pod \"prometheus-k8s-0\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:23.264114 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:23.263598    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:26.258573 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:26.248420    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 13:07:26.307960 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:26.306497    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert\") pod \"ingress-canary-7v2wh\" (UID: \"8b9092df-01f8-4d59-82f8-a058e84b55f5\") " pod="openshift-ingress-canary/ingress-canary-7v2wh"
May 25 13:07:26.307960 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:26.306548    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls\") pod \"dns-default-27ppb\" (UID: \"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293\") " pod="openshift-dns/dns-default-27ppb"
May 25 13:07:26.307960 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:26.306620    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:07:26.310093 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:26.310067    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls\") pod \"image-registry-84ccf98695-7zpjl\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") " pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:07:26.312182 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:26.312098    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0b340d4-ffa3-4bf8-aeda-9d13cf88d293-metrics-tls\") pod \"dns-default-27ppb\" (UID: \"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293\") " pod="openshift-dns/dns-default-27ppb"
May 25 13:07:26.313991 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:26.313948    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b9092df-01f8-4d59-82f8-a058e84b55f5-cert\") pod \"ingress-canary-7v2wh\" (UID: \"8b9092df-01f8-4d59-82f8-a058e84b55f5\") " pod="openshift-ingress-canary/ingress-canary-7v2wh"
May 25 13:07:26.580454 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:26.580227    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zhtj8\""
May 25 13:07:26.580454 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:26.580284    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9wl44\""
May 25 13:07:26.580454 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:26.580353    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6tjlh\""
May 25 13:07:26.588600 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:26.588253    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:07:26.588744 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:26.588621    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7v2wh"
May 25 13:07:26.588744 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:26.588671    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-27ppb"
May 25 13:07:26.856157 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:26.856102    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7v2wh"]
May 25 13:07:26.858758 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:07:26.858694    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9092df_01f8_4d59_82f8_a058e84b55f5.slice/crio-ffef5c8f42948b2fa0e388500a11031a579a379d101e38886eb51d372fcc8954 WatchSource:0}: Error finding container ffef5c8f42948b2fa0e388500a11031a579a379d101e38886eb51d372fcc8954: Status 404 returned error can't find the container with id ffef5c8f42948b2fa0e388500a11031a579a379d101e38886eb51d372fcc8954
May 25 13:07:26.874559 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:26.874346    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-27ppb"]
May 25 13:07:26.879705 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:07:26.879663    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0b340d4_ffa3_4bf8_aeda_9d13cf88d293.slice/crio-84ed49a6918bbd9540ee02b3bab87b11dc7cbfca53caf735864a733c7a58402d WatchSource:0}: Error finding container 84ed49a6918bbd9540ee02b3bab87b11dc7cbfca53caf735864a733c7a58402d: Status 404 returned error can't find the container with id 84ed49a6918bbd9540ee02b3bab87b11dc7cbfca53caf735864a733c7a58402d
May 25 13:07:26.883671 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:26.883222    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84ccf98695-7zpjl"]
May 25 13:07:26.887299 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:07:26.887266    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b30c873_1845_484c_ae06_b3eab38d772d.slice/crio-9ed9064f2f2e20a75af492cbad40a75adcdd0362392825bae3434fc19c935fcd WatchSource:0}: Error finding container 9ed9064f2f2e20a75af492cbad40a75adcdd0362392825bae3434fc19c935fcd: Status 404 returned error can't find the container with id 9ed9064f2f2e20a75af492cbad40a75adcdd0362392825bae3434fc19c935fcd
May 25 13:07:27.105910 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:27.105868    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84ccf98695-7zpjl" event={"ID":"4b30c873-1845-484c-ae06-b3eab38d772d","Type":"ContainerStarted","Data":"6879184f09ebe21e928a3edcf4d28eecb0cd3702d998d8a30f7907418e51fc87"}
May 25 13:07:27.105910 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:27.105917    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84ccf98695-7zpjl" event={"ID":"4b30c873-1845-484c-ae06-b3eab38d772d","Type":"ContainerStarted","Data":"9ed9064f2f2e20a75af492cbad40a75adcdd0362392825bae3434fc19c935fcd"}
May 25 13:07:27.106290 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:27.106047    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:07:27.109961 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:27.109899    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8644f8cf75-c69k8" event={"ID":"bff5617f-f53f-4d47-afb9-4b68d3b159ed","Type":"ContainerStarted","Data":"281e1f376b5f9c99903a9b6123e11ef342a6bf86bb54ced930924b1882667c9c"}
May 25 13:07:27.114869 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:27.114828    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x7b5z" event={"ID":"f1152b71-6f95-4a41-87e0-98fccf6c5512","Type":"ContainerStarted","Data":"f1df0b1ea93e4bee431a7fb8d8f3b07d9ac69ce025e1dffaf83c9ea7581b88bf"}
May 25 13:07:27.115044 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:27.114881    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x7b5z" event={"ID":"f1152b71-6f95-4a41-87e0-98fccf6c5512","Type":"ContainerStarted","Data":"543334242c32ea36b1b287269eb0ef34ee9e932a1db4545286716a2a4e2aa234"}
May 25 13:07:27.121500 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:27.120737    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-8w47r" event={"ID":"b8e9b997-2768-47e6-82fb-5f498706590e","Type":"ContainerStarted","Data":"c3185535630b4c388c5aaa47d997a83ac4bc590ee69c0ef0d1a90c812f5d9310"}
May 25 13:07:27.123278 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:27.123219    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-27ppb" event={"ID":"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293","Type":"ContainerStarted","Data":"84ed49a6918bbd9540ee02b3bab87b11dc7cbfca53caf735864a733c7a58402d"}
May 25 13:07:27.129902 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:27.129842    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84ccf98695-7zpjl" podStartSLOduration=162.12982348 podStartE2EDuration="2m42.12982348s" podCreationTimestamp="2026-05-25 13:04:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:07:27.129625782 +0000 UTC m=+163.132367537" watchObservedRunningTime="2026-05-25 13:07:27.12982348 +0000 UTC m=+163.132565236"
May 25 13:07:27.132098 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:27.132057    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn" event={"ID":"5109856c-fa52-4fc8-855d-ca1dff292b04","Type":"ContainerStarted","Data":"a2a863b07819c4358c325c38d192d1a5849a158d9fe6b563fbae35ded9c12275"}
May 25 13:07:27.132307 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:27.132116    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn" event={"ID":"5109856c-fa52-4fc8-855d-ca1dff292b04","Type":"ContainerStarted","Data":"74da69695652760342b5a9ab83015d7a586e9054ed9101720d725ea5f33801d5"}
May 25 13:07:27.132307 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:27.132130    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn" event={"ID":"5109856c-fa52-4fc8-855d-ca1dff292b04","Type":"ContainerStarted","Data":"f9594aab9aabe942cf02c81b5a64465c5b2b9882065392c9acc4d641d1ffb227"}
May 25 13:07:27.140122 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:27.140089    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d" event={"ID":"ff44f516-493d-4bab-8b9c-841c79d799a7","Type":"ContainerStarted","Data":"cd1e7478d914ee15770de3f76118d9576d4d6fb9d8c188f6ab09824eaff7bd11"}
May 25 13:07:27.140405 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:27.140383    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d" event={"ID":"ff44f516-493d-4bab-8b9c-841c79d799a7","Type":"ContainerStarted","Data":"1c1a347f9dbacab556a525273175c02825335a10a6eb957ac8c93f61819d8759"}
May 25 13:07:27.140541 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:27.140526    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d" event={"ID":"ff44f516-493d-4bab-8b9c-841c79d799a7","Type":"ContainerStarted","Data":"e7afeb883ead45df5d9dd83640b1a129fc99229fe11471e47ba5d2b579040e29"}
May 25 13:07:27.141961 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:27.141930    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4","Type":"ContainerStarted","Data":"a794c28985528d2e21702bc8c898c9944a854ca6e1f1b10a2b70f2477ad9269f"}
May 25 13:07:27.143735 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:27.143688    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7v2wh" event={"ID":"8b9092df-01f8-4d59-82f8-a058e84b55f5","Type":"ContainerStarted","Data":"ffef5c8f42948b2fa0e388500a11031a579a379d101e38886eb51d372fcc8954"}
May 25 13:07:27.153923 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:27.153817    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-x7b5z" podStartSLOduration=9.588829417 podStartE2EDuration="11.153793049s" podCreationTimestamp="2026-05-25 13:07:16 +0000 UTC" firstStartedPulling="2026-05-25 13:07:19.930459617 +0000 UTC m=+155.933201354" lastFinishedPulling="2026-05-25 13:07:21.495423252 +0000 UTC m=+157.498164986" observedRunningTime="2026-05-25 13:07:27.151976278 +0000 UTC m=+163.154718032" watchObservedRunningTime="2026-05-25 13:07:27.153793049 +0000 UTC m=+163.156534803"
May 25 13:07:27.172086 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:27.172019    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-8644f8cf75-c69k8" podStartSLOduration=8.186520433 podStartE2EDuration="14.171999875s" podCreationTimestamp="2026-05-25 13:07:13 +0000 UTC" firstStartedPulling="2026-05-25 13:07:20.079821888 +0000 UTC m=+156.082563633" lastFinishedPulling="2026-05-25 13:07:26.065301325 +0000 UTC m=+162.068043075" observedRunningTime="2026-05-25 13:07:27.169947811 +0000 UTC m=+163.172689568" watchObservedRunningTime="2026-05-25 13:07:27.171999875 +0000 UTC m=+163.174741631"
May 25 13:07:27.188893 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:27.188824    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-8w47r" podStartSLOduration=5.604561125 podStartE2EDuration="11.18880489s" podCreationTimestamp="2026-05-25 13:07:16 +0000 UTC" firstStartedPulling="2026-05-25 13:07:20.470093858 +0000 UTC m=+156.472835593" lastFinishedPulling="2026-05-25 13:07:26.054337611 +0000 UTC m=+162.057079358" observedRunningTime="2026-05-25 13:07:27.187462465 +0000 UTC m=+163.190204221" watchObservedRunningTime="2026-05-25 13:07:27.18880489 +0000 UTC m=+163.191546646"
May 25 13:07:27.210732 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:27.210239    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-znb4d" podStartSLOduration=5.463341886 podStartE2EDuration="11.210215994s" podCreationTimestamp="2026-05-25 13:07:16 +0000 UTC" firstStartedPulling="2026-05-25 13:07:20.309242413 +0000 UTC m=+156.311984165" lastFinishedPulling="2026-05-25 13:07:26.056116542 +0000 UTC m=+162.058858273" observedRunningTime="2026-05-25 13:07:27.208043334 +0000 UTC m=+163.210785101" watchObservedRunningTime="2026-05-25 13:07:27.210215994 +0000 UTC m=+163.212957750"
May 25 13:07:30.158788 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:30.158752    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn" event={"ID":"5109856c-fa52-4fc8-855d-ca1dff292b04","Type":"ContainerStarted","Data":"c4a786f1b1faf7f8a6888e46bd2bf52cbee6b1d1b65a0b1797dcb565fd915f0f"}
May 25 13:07:30.159288 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:30.158800    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn" event={"ID":"5109856c-fa52-4fc8-855d-ca1dff292b04","Type":"ContainerStarted","Data":"739d95602f31cb9412a86a506ab69cac031228694c41ce85a2e9391abb84a751"}
May 25 13:07:30.160362 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:30.160334    2569 generic.go:358] "Generic (PLEG): container finished" podID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerID="915c5c6308a52cb75efa1900a479c023ad49bb4e1b3aea398d9665efd4d0360e" exitCode=0
May 25 13:07:30.160466 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:30.160436    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4","Type":"ContainerDied","Data":"915c5c6308a52cb75efa1900a479c023ad49bb4e1b3aea398d9665efd4d0360e"}
May 25 13:07:30.162517 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:30.162482    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7v2wh" event={"ID":"8b9092df-01f8-4d59-82f8-a058e84b55f5","Type":"ContainerStarted","Data":"de6fd32b88001316f3e8a2e12f248be7a3a20f86bbe969f54759c59c062d7ecf"}
May 25 13:07:30.170900 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:30.167410    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-27ppb" event={"ID":"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293","Type":"ContainerStarted","Data":"503e19b490aa97a1a19dc3b193ed3947c7f97184147ebeaab95a6763c2fb90ca"}
May 25 13:07:30.206365 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:30.206313    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7v2wh" podStartSLOduration=129.17389007 podStartE2EDuration="2m12.206296362s" podCreationTimestamp="2026-05-25 13:05:18 +0000 UTC" firstStartedPulling="2026-05-25 13:07:26.862312852 +0000 UTC m=+162.865054601" lastFinishedPulling="2026-05-25 13:07:29.894719158 +0000 UTC m=+165.897460893" observedRunningTime="2026-05-25 13:07:30.205473428 +0000 UTC m=+166.208215179" watchObservedRunningTime="2026-05-25 13:07:30.206296362 +0000 UTC m=+166.209038116"
May 25 13:07:31.173789 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:31.173698    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn" event={"ID":"5109856c-fa52-4fc8-855d-ca1dff292b04","Type":"ContainerStarted","Data":"660a93e849fe35cec1908dbd13a39e36b4da42307f32669fb47b9657cb0bd85d"}
May 25 13:07:31.174272 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:31.174000    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:31.175611 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:31.175574    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-27ppb" event={"ID":"d0b340d4-ffa3-4bf8-aeda-9d13cf88d293","Type":"ContainerStarted","Data":"e6cdebdb0be0dca7bf22c11e49b51931a4185f8c54699dc1d39df0437c29ce09"}
May 25 13:07:31.181181 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:31.181147    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn"
May 25 13:07:31.196519 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:31.196469    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-75d748b86c-j78qn" podStartSLOduration=2.71199311 podStartE2EDuration="12.196447824s" podCreationTimestamp="2026-05-25 13:07:19 +0000 UTC" firstStartedPulling="2026-05-25 13:07:20.191837641 +0000 UTC m=+156.194579373" lastFinishedPulling="2026-05-25 13:07:29.676292355 +0000 UTC m=+165.679034087" observedRunningTime="2026-05-25 13:07:31.196277739 +0000 UTC m=+167.199019492" watchObservedRunningTime="2026-05-25 13:07:31.196447824 +0000 UTC m=+167.199189580"
May 25 13:07:31.212047 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:31.211987    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-27ppb" podStartSLOduration=130.197250601 podStartE2EDuration="2m13.21196964s" podCreationTimestamp="2026-05-25 13:05:18 +0000 UTC" firstStartedPulling="2026-05-25 13:07:26.883316078 +0000 UTC m=+162.886057825" lastFinishedPulling="2026-05-25 13:07:29.898035132 +0000 UTC m=+165.900776864" observedRunningTime="2026-05-25 13:07:31.211441408 +0000 UTC m=+167.214183163" watchObservedRunningTime="2026-05-25 13:07:31.21196964 +0000 UTC m=+167.214711395"
May 25 13:07:32.180267 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:32.180206    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-27ppb"
May 25 13:07:33.597677 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:33.597306    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:07:33.725008 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:33.724968    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:34.189653 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:34.189574    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4","Type":"ContainerStarted","Data":"e1a2c9f12a192a69504fd7b7e2140f87078af39c219ebe78a6efedd28b27526c"}
May 25 13:07:34.189653 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:34.189613    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4","Type":"ContainerStarted","Data":"96b6458cd0c6dde2dc4cda0c284fcbb80c327d882f90c8bf3cf19e9a124a588d"}
May 25 13:07:34.189653 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:34.189623    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4","Type":"ContainerStarted","Data":"bf3c81701e53026ca856ca50846a11fcb5aa23b78110c7e404d30dd881140bd2"}
May 25 13:07:34.189653 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:34.189633    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4","Type":"ContainerStarted","Data":"8467305e0f33c532e4bca8ea94cdffbbf30f129d0d219237673a2e4e64538c53"}
May 25 13:07:34.189653 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:34.189645    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4","Type":"ContainerStarted","Data":"5408f6a405b115e05f8413691be2621edf156bc52cf81b0ec71e52670c5f68fe"}
May 25 13:07:34.189907 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:34.189656    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4","Type":"ContainerStarted","Data":"161784d54901e7c3d063ce6c39404cab72424acdf62c96035ae41865c7763ccd"}
May 25 13:07:34.219035 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:34.218975    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.143225042 podStartE2EDuration="12.218957223s" podCreationTimestamp="2026-05-25 13:07:22 +0000 UTC" firstStartedPulling="2026-05-25 13:07:26.271145696 +0000 UTC m=+162.273887433" lastFinishedPulling="2026-05-25 13:07:33.346877868 +0000 UTC m=+169.349619614" observedRunningTime="2026-05-25 13:07:34.216757861 +0000 UTC m=+170.219499614" watchObservedRunningTime="2026-05-25 13:07:34.218957223 +0000 UTC m=+170.221698978"
May 25 13:07:38.264414 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:38.264374    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:07:42.185602 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:42.185570    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-27ppb"
May 25 13:07:46.592986 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:46.592927    2569 patch_prober.go:28] interesting pod/image-registry-84ccf98695-7zpjl container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 25 13:07:46.593477 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:46.593032    2569 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-84ccf98695-7zpjl" podUID="4b30c873-1845-484c-ae06-b3eab38d772d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:07:48.151910 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:48.151880    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:07:52.138829 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:52.138757    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-8644f8cf75-c69k8" podUID="bff5617f-f53f-4d47-afb9-4b68d3b159ed" containerName="console" containerID="cri-o://281e1f376b5f9c99903a9b6123e11ef342a6bf86bb54ced930924b1882667c9c" gracePeriod=15
May 25 13:07:52.393595 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:52.393518    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8644f8cf75-c69k8_bff5617f-f53f-4d47-afb9-4b68d3b159ed/console/0.log"
May 25 13:07:52.393720 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:52.393608    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:52.456662 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:52.456620    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bff5617f-f53f-4d47-afb9-4b68d3b159ed-oauth-serving-cert\") pod \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") "
May 25 13:07:52.456854 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:52.456673    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4fvr\" (UniqueName: \"kubernetes.io/projected/bff5617f-f53f-4d47-afb9-4b68d3b159ed-kube-api-access-l4fvr\") pod \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") "
May 25 13:07:52.456854 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:52.456699    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bff5617f-f53f-4d47-afb9-4b68d3b159ed-console-config\") pod \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") "
May 25 13:07:52.456854 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:52.456773    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bff5617f-f53f-4d47-afb9-4b68d3b159ed-console-serving-cert\") pod \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") "
May 25 13:07:52.457029 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:52.456907    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bff5617f-f53f-4d47-afb9-4b68d3b159ed-service-ca\") pod \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") "
May 25 13:07:52.457029 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:52.457021    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bff5617f-f53f-4d47-afb9-4b68d3b159ed-console-oauth-config\") pod \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\" (UID: \"bff5617f-f53f-4d47-afb9-4b68d3b159ed\") "
May 25 13:07:52.457136 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:52.457094    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff5617f-f53f-4d47-afb9-4b68d3b159ed-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bff5617f-f53f-4d47-afb9-4b68d3b159ed" (UID: "bff5617f-f53f-4d47-afb9-4b68d3b159ed"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:07:52.457260 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:52.457229    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff5617f-f53f-4d47-afb9-4b68d3b159ed-console-config" (OuterVolumeSpecName: "console-config") pod "bff5617f-f53f-4d47-afb9-4b68d3b159ed" (UID: "bff5617f-f53f-4d47-afb9-4b68d3b159ed"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:07:52.457404 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:52.457277    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff5617f-f53f-4d47-afb9-4b68d3b159ed-service-ca" (OuterVolumeSpecName: "service-ca") pod "bff5617f-f53f-4d47-afb9-4b68d3b159ed" (UID: "bff5617f-f53f-4d47-afb9-4b68d3b159ed"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:07:52.457404 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:52.457371    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bff5617f-f53f-4d47-afb9-4b68d3b159ed-oauth-serving-cert\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:07:52.457404 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:52.457393    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bff5617f-f53f-4d47-afb9-4b68d3b159ed-console-config\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:07:52.457545 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:52.457406    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bff5617f-f53f-4d47-afb9-4b68d3b159ed-service-ca\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:07:52.459035 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:52.459012    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bff5617f-f53f-4d47-afb9-4b68d3b159ed-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bff5617f-f53f-4d47-afb9-4b68d3b159ed" (UID: "bff5617f-f53f-4d47-afb9-4b68d3b159ed"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:07:52.459108 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:52.459042    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bff5617f-f53f-4d47-afb9-4b68d3b159ed-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bff5617f-f53f-4d47-afb9-4b68d3b159ed" (UID: "bff5617f-f53f-4d47-afb9-4b68d3b159ed"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:07:52.459145 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:52.459126    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff5617f-f53f-4d47-afb9-4b68d3b159ed-kube-api-access-l4fvr" (OuterVolumeSpecName: "kube-api-access-l4fvr") pod "bff5617f-f53f-4d47-afb9-4b68d3b159ed" (UID: "bff5617f-f53f-4d47-afb9-4b68d3b159ed"). InnerVolumeSpecName "kube-api-access-l4fvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:07:52.557896 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:52.557851    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4fvr\" (UniqueName: \"kubernetes.io/projected/bff5617f-f53f-4d47-afb9-4b68d3b159ed-kube-api-access-l4fvr\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:07:52.557896 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:52.557884    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bff5617f-f53f-4d47-afb9-4b68d3b159ed-console-serving-cert\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:07:52.557896 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:52.557894    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bff5617f-f53f-4d47-afb9-4b68d3b159ed-console-oauth-config\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:07:53.250880 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:53.250852    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8644f8cf75-c69k8_bff5617f-f53f-4d47-afb9-4b68d3b159ed/console/0.log"
May 25 13:07:53.251417 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:53.250893    2569 generic.go:358] "Generic (PLEG): container finished" podID="bff5617f-f53f-4d47-afb9-4b68d3b159ed" containerID="281e1f376b5f9c99903a9b6123e11ef342a6bf86bb54ced930924b1882667c9c" exitCode=2
May 25 13:07:53.251417 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:53.250925    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8644f8cf75-c69k8" event={"ID":"bff5617f-f53f-4d47-afb9-4b68d3b159ed","Type":"ContainerDied","Data":"281e1f376b5f9c99903a9b6123e11ef342a6bf86bb54ced930924b1882667c9c"}
May 25 13:07:53.251417 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:53.250964    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8644f8cf75-c69k8" event={"ID":"bff5617f-f53f-4d47-afb9-4b68d3b159ed","Type":"ContainerDied","Data":"4c51b3ad605670c893eb791dd94fcc037454b96a91a0b5460684a1937c235b2e"}
May 25 13:07:53.251417 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:53.250979    2569 scope.go:117] "RemoveContainer" containerID="281e1f376b5f9c99903a9b6123e11ef342a6bf86bb54ced930924b1882667c9c"
May 25 13:07:53.251417 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:53.250981    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8644f8cf75-c69k8"
May 25 13:07:53.259245 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:53.259223    2569 scope.go:117] "RemoveContainer" containerID="281e1f376b5f9c99903a9b6123e11ef342a6bf86bb54ced930924b1882667c9c"
May 25 13:07:53.259558 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:07:53.259538    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281e1f376b5f9c99903a9b6123e11ef342a6bf86bb54ced930924b1882667c9c\": container with ID starting with 281e1f376b5f9c99903a9b6123e11ef342a6bf86bb54ced930924b1882667c9c not found: ID does not exist" containerID="281e1f376b5f9c99903a9b6123e11ef342a6bf86bb54ced930924b1882667c9c"
May 25 13:07:53.259604 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:53.259567    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281e1f376b5f9c99903a9b6123e11ef342a6bf86bb54ced930924b1882667c9c"} err="failed to get container status \"281e1f376b5f9c99903a9b6123e11ef342a6bf86bb54ced930924b1882667c9c\": rpc error: code = NotFound desc = could not find container \"281e1f376b5f9c99903a9b6123e11ef342a6bf86bb54ced930924b1882667c9c\": container with ID starting with 281e1f376b5f9c99903a9b6123e11ef342a6bf86bb54ced930924b1882667c9c not found: ID does not exist"
May 25 13:07:53.268890 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:53.268861    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8644f8cf75-c69k8"]
May 25 13:07:53.270283 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:53.270264    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8644f8cf75-c69k8"]
May 25 13:07:53.352288 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:53.352253    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84ccf98695-7zpjl"]
May 25 13:07:54.597639 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:07:54.597598    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bff5617f-f53f-4d47-afb9-4b68d3b159ed" path="/var/lib/kubelet/pods/bff5617f-f53f-4d47-afb9-4b68d3b159ed/volumes"
May 25 13:08:04.285204 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:04.285144    2569 generic.go:358] "Generic (PLEG): container finished" podID="0d406e4f-9b85-403f-b250-242f6d50a43f" containerID="7c71d7c85714d972d7e5f9bf0681997268f6c7e3f4306b94904bfd503bc58f04" exitCode=0
May 25 13:08:04.285620 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:04.285214    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7spmp" event={"ID":"0d406e4f-9b85-403f-b250-242f6d50a43f","Type":"ContainerDied","Data":"7c71d7c85714d972d7e5f9bf0681997268f6c7e3f4306b94904bfd503bc58f04"}
May 25 13:08:04.285620 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:04.285569    2569 scope.go:117] "RemoveContainer" containerID="7c71d7c85714d972d7e5f9bf0681997268f6c7e3f4306b94904bfd503bc58f04"
May 25 13:08:05.290580 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:05.290540    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7spmp" event={"ID":"0d406e4f-9b85-403f-b250-242f6d50a43f","Type":"ContainerStarted","Data":"cf307906b406339f23609f81b51ba5cccc10c3d03ca9be8cafeca74dd88e42fb"}
May 25 13:08:18.372779 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:18.372717    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84ccf98695-7zpjl" podUID="4b30c873-1845-484c-ae06-b3eab38d772d" containerName="registry" containerID="cri-o://6879184f09ebe21e928a3edcf4d28eecb0cd3702d998d8a30f7907418e51fc87" gracePeriod=30
May 25 13:08:19.615514 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.615489    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:08:19.801839 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.801729    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vz76w\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-kube-api-access-vz76w\") pod \"4b30c873-1845-484c-ae06-b3eab38d772d\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") "
May 25 13:08:19.801839 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.801787    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls\") pod \"4b30c873-1845-484c-ae06-b3eab38d772d\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") "
May 25 13:08:19.801839 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.801810    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-bound-sa-token\") pod \"4b30c873-1845-484c-ae06-b3eab38d772d\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") "
May 25 13:08:19.802129 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.801859    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b30c873-1845-484c-ae06-b3eab38d772d-trusted-ca\") pod \"4b30c873-1845-484c-ae06-b3eab38d772d\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") "
May 25 13:08:19.802129 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.801956    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4b30c873-1845-484c-ae06-b3eab38d772d-registry-certificates\") pod \"4b30c873-1845-484c-ae06-b3eab38d772d\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") "
May 25 13:08:19.802129 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.802015    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4b30c873-1845-484c-ae06-b3eab38d772d-ca-trust-extracted\") pod \"4b30c873-1845-484c-ae06-b3eab38d772d\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") "
May 25 13:08:19.802129 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.802059    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4b30c873-1845-484c-ae06-b3eab38d772d-image-registry-private-configuration\") pod \"4b30c873-1845-484c-ae06-b3eab38d772d\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") "
May 25 13:08:19.802129 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.802084    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4b30c873-1845-484c-ae06-b3eab38d772d-installation-pull-secrets\") pod \"4b30c873-1845-484c-ae06-b3eab38d772d\" (UID: \"4b30c873-1845-484c-ae06-b3eab38d772d\") "
May 25 13:08:19.802424 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.802355    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b30c873-1845-484c-ae06-b3eab38d772d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4b30c873-1845-484c-ae06-b3eab38d772d" (UID: "4b30c873-1845-484c-ae06-b3eab38d772d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:08:19.802557 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.802512    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b30c873-1845-484c-ae06-b3eab38d772d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4b30c873-1845-484c-ae06-b3eab38d772d" (UID: "4b30c873-1845-484c-ae06-b3eab38d772d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:08:19.804580 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.804518    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4b30c873-1845-484c-ae06-b3eab38d772d" (UID: "4b30c873-1845-484c-ae06-b3eab38d772d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:08:19.804580 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.804570    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b30c873-1845-484c-ae06-b3eab38d772d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4b30c873-1845-484c-ae06-b3eab38d772d" (UID: "4b30c873-1845-484c-ae06-b3eab38d772d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:08:19.804779 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.804679    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-kube-api-access-vz76w" (OuterVolumeSpecName: "kube-api-access-vz76w") pod "4b30c873-1845-484c-ae06-b3eab38d772d" (UID: "4b30c873-1845-484c-ae06-b3eab38d772d"). InnerVolumeSpecName "kube-api-access-vz76w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:08:19.804843 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.804805    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4b30c873-1845-484c-ae06-b3eab38d772d" (UID: "4b30c873-1845-484c-ae06-b3eab38d772d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:08:19.804895 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.804880    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b30c873-1845-484c-ae06-b3eab38d772d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4b30c873-1845-484c-ae06-b3eab38d772d" (UID: "4b30c873-1845-484c-ae06-b3eab38d772d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:08:19.811319 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.811289    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b30c873-1845-484c-ae06-b3eab38d772d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4b30c873-1845-484c-ae06-b3eab38d772d" (UID: "4b30c873-1845-484c-ae06-b3eab38d772d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:08:19.903682 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.903642    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz76w\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-kube-api-access-vz76w\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:19.903682 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.903675    2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-registry-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:19.903682 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.903686    2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b30c873-1845-484c-ae06-b3eab38d772d-bound-sa-token\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:19.903682 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.903695    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b30c873-1845-484c-ae06-b3eab38d772d-trusted-ca\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:19.903943 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.903703    2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4b30c873-1845-484c-ae06-b3eab38d772d-registry-certificates\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:19.903943 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.903712    2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4b30c873-1845-484c-ae06-b3eab38d772d-ca-trust-extracted\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:19.903943 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.903722    2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4b30c873-1845-484c-ae06-b3eab38d772d-image-registry-private-configuration\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:19.903943 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:19.903731    2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4b30c873-1845-484c-ae06-b3eab38d772d-installation-pull-secrets\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:20.342739 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:20.342700    2569 generic.go:358] "Generic (PLEG): container finished" podID="4b30c873-1845-484c-ae06-b3eab38d772d" containerID="6879184f09ebe21e928a3edcf4d28eecb0cd3702d998d8a30f7907418e51fc87" exitCode=0
May 25 13:08:20.342937 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:20.342771    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84ccf98695-7zpjl" event={"ID":"4b30c873-1845-484c-ae06-b3eab38d772d","Type":"ContainerDied","Data":"6879184f09ebe21e928a3edcf4d28eecb0cd3702d998d8a30f7907418e51fc87"}
May 25 13:08:20.342937 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:20.342790    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84ccf98695-7zpjl"
May 25 13:08:20.342937 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:20.342816    2569 scope.go:117] "RemoveContainer" containerID="6879184f09ebe21e928a3edcf4d28eecb0cd3702d998d8a30f7907418e51fc87"
May 25 13:08:20.342937 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:20.342804    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84ccf98695-7zpjl" event={"ID":"4b30c873-1845-484c-ae06-b3eab38d772d","Type":"ContainerDied","Data":"9ed9064f2f2e20a75af492cbad40a75adcdd0362392825bae3434fc19c935fcd"}
May 25 13:08:20.351320 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:20.351296    2569 scope.go:117] "RemoveContainer" containerID="6879184f09ebe21e928a3edcf4d28eecb0cd3702d998d8a30f7907418e51fc87"
May 25 13:08:20.351642 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:08:20.351617    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6879184f09ebe21e928a3edcf4d28eecb0cd3702d998d8a30f7907418e51fc87\": container with ID starting with 6879184f09ebe21e928a3edcf4d28eecb0cd3702d998d8a30f7907418e51fc87 not found: ID does not exist" containerID="6879184f09ebe21e928a3edcf4d28eecb0cd3702d998d8a30f7907418e51fc87"
May 25 13:08:20.351721 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:20.351650    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6879184f09ebe21e928a3edcf4d28eecb0cd3702d998d8a30f7907418e51fc87"} err="failed to get container status \"6879184f09ebe21e928a3edcf4d28eecb0cd3702d998d8a30f7907418e51fc87\": rpc error: code = NotFound desc = could not find container \"6879184f09ebe21e928a3edcf4d28eecb0cd3702d998d8a30f7907418e51fc87\": container with ID starting with 6879184f09ebe21e928a3edcf4d28eecb0cd3702d998d8a30f7907418e51fc87 not found: ID does not exist"
May 25 13:08:20.364254 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:20.364205    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84ccf98695-7zpjl"]
May 25 13:08:20.368126 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:20.368098    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84ccf98695-7zpjl"]
May 25 13:08:20.597376 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:20.597343    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b30c873-1845-484c-ae06-b3eab38d772d" path="/var/lib/kubelet/pods/4b30c873-1845-484c-ae06-b3eab38d772d/volumes"
May 25 13:08:23.264509 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:23.264457    2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:23.283339 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:23.283311    2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:23.367371 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:23.367345    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:41.282091 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.282056    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 13:08:41.282614 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.282532    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="prometheus" containerID="cri-o://161784d54901e7c3d063ce6c39404cab72424acdf62c96035ae41865c7763ccd" gracePeriod=600
May 25 13:08:41.282614 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.282550    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="kube-rbac-proxy" containerID="cri-o://96b6458cd0c6dde2dc4cda0c284fcbb80c327d882f90c8bf3cf19e9a124a588d" gracePeriod=600
May 25 13:08:41.282614 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.282574    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="thanos-sidecar" containerID="cri-o://8467305e0f33c532e4bca8ea94cdffbbf30f129d0d219237673a2e4e64538c53" gracePeriod=600
May 25 13:08:41.282782 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.282647    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e1a2c9f12a192a69504fd7b7e2140f87078af39c219ebe78a6efedd28b27526c" gracePeriod=600
May 25 13:08:41.282782 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.282575    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="config-reloader" containerID="cri-o://5408f6a405b115e05f8413691be2621edf156bc52cf81b0ec71e52670c5f68fe" gracePeriod=600
May 25 13:08:41.282782 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.282744    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="kube-rbac-proxy-web" containerID="cri-o://bf3c81701e53026ca856ca50846a11fcb5aa23b78110c7e404d30dd881140bd2" gracePeriod=600
May 25 13:08:41.411431 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.411402    2569 generic.go:358] "Generic (PLEG): container finished" podID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerID="e1a2c9f12a192a69504fd7b7e2140f87078af39c219ebe78a6efedd28b27526c" exitCode=0
May 25 13:08:41.411431 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.411429    2569 generic.go:358] "Generic (PLEG): container finished" podID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerID="96b6458cd0c6dde2dc4cda0c284fcbb80c327d882f90c8bf3cf19e9a124a588d" exitCode=0
May 25 13:08:41.411431 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.411437    2569 generic.go:358] "Generic (PLEG): container finished" podID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerID="bf3c81701e53026ca856ca50846a11fcb5aa23b78110c7e404d30dd881140bd2" exitCode=0
May 25 13:08:41.411431 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.411445    2569 generic.go:358] "Generic (PLEG): container finished" podID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerID="8467305e0f33c532e4bca8ea94cdffbbf30f129d0d219237673a2e4e64538c53" exitCode=0
May 25 13:08:41.411675 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.411453    2569 generic.go:358] "Generic (PLEG): container finished" podID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerID="5408f6a405b115e05f8413691be2621edf156bc52cf81b0ec71e52670c5f68fe" exitCode=0
May 25 13:08:41.411675 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.411461    2569 generic.go:358] "Generic (PLEG): container finished" podID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerID="161784d54901e7c3d063ce6c39404cab72424acdf62c96035ae41865c7763ccd" exitCode=0
May 25 13:08:41.411675 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.411473    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4","Type":"ContainerDied","Data":"e1a2c9f12a192a69504fd7b7e2140f87078af39c219ebe78a6efedd28b27526c"}
May 25 13:08:41.411675 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.411512    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4","Type":"ContainerDied","Data":"96b6458cd0c6dde2dc4cda0c284fcbb80c327d882f90c8bf3cf19e9a124a588d"}
May 25 13:08:41.411675 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.411522    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4","Type":"ContainerDied","Data":"bf3c81701e53026ca856ca50846a11fcb5aa23b78110c7e404d30dd881140bd2"}
May 25 13:08:41.411675 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.411531    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4","Type":"ContainerDied","Data":"8467305e0f33c532e4bca8ea94cdffbbf30f129d0d219237673a2e4e64538c53"}
May 25 13:08:41.411675 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.411539    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4","Type":"ContainerDied","Data":"5408f6a405b115e05f8413691be2621edf156bc52cf81b0ec71e52670c5f68fe"}
May 25 13:08:41.411675 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.411547    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4","Type":"ContainerDied","Data":"161784d54901e7c3d063ce6c39404cab72424acdf62c96035ae41865c7763ccd"}
May 25 13:08:41.544972 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.544947    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:41.580343 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.580313    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-prometheus-trusted-ca-bundle\") pod \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") "
May 25 13:08:41.580514 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.580359    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") "
May 25 13:08:41.580514 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.580389    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") "
May 25 13:08:41.580514 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.580409    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-configmap-metrics-client-ca\") pod \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") "
May 25 13:08:41.580514 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.580433    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-thanos-prometheus-http-client-file\") pod \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") "
May 25 13:08:41.580514 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.580466    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-tls-assets\") pod \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") "
May 25 13:08:41.580514 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.580500    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-kube-rbac-proxy\") pod \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") "
May 25 13:08:41.580822 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.580524    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-configmap-kubelet-serving-ca-bundle\") pod \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") "
May 25 13:08:41.580877 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.580853    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" (UID: "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:08:41.580930 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.580905    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" (UID: "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:08:41.581000 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.580981    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-web-config\") pod \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") "
May 25 13:08:41.581061 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.581041    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vlzb\" (UniqueName: \"kubernetes.io/projected/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-kube-api-access-8vlzb\") pod \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") "
May 25 13:08:41.581114 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.581079    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-grpc-tls\") pod \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") "
May 25 13:08:41.581114 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.581105    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-config-out\") pod \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") "
May 25 13:08:41.581233 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.581138    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-prometheus-k8s-rulefiles-0\") pod \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") "
May 25 13:08:41.581233 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.581200    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-configmap-serving-certs-ca-bundle\") pod \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") "
May 25 13:08:41.581337 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.581239    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-prometheus-k8s-tls\") pod \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") "
May 25 13:08:41.581599 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.581389    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-prometheus-k8s-db\") pod \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") "
May 25 13:08:41.581599 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.581420    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-metrics-client-certs\") pod \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") "
May 25 13:08:41.581599 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.581477    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-config\") pod \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\" (UID: \"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4\") "
May 25 13:08:41.581813 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.581633    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" (UID: "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:08:41.581813 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.581798    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:41.581933 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.581818    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-configmap-metrics-client-ca\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:41.581933 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.581834    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:41.583472 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.583320    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" (UID: "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:08:41.584327 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.584294    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" (UID: "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:08:41.584438 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.584382    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-config" (OuterVolumeSpecName: "config") pod "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" (UID: "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:08:41.584801 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.584780    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" (UID: "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:08:41.585228 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.585189    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" (UID: "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:08:41.585719 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.585685    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" (UID: "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:08:41.585902 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.585865    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" (UID: "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:08:41.587917 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.587225    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" (UID: "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:08:41.587917 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.587301    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" (UID: "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:08:41.587917 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.587311    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-config-out" (OuterVolumeSpecName: "config-out") pod "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" (UID: "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:08:41.587917 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.587617    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" (UID: "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:08:41.587917 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.587779    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" (UID: "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:08:41.587917 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.587865    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" (UID: "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:08:41.588314 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.587968    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-kube-api-access-8vlzb" (OuterVolumeSpecName: "kube-api-access-8vlzb") pod "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" (UID: "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4"). InnerVolumeSpecName "kube-api-access-8vlzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:08:41.597536 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.597507    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-web-config" (OuterVolumeSpecName: "web-config") pod "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" (UID: "9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:08:41.683125 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.683074    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:41.683125 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.683118    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:41.683125 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.683130    2569 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:41.683384 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.683140    2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-tls-assets\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:41.683384 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.683151    2569 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-kube-rbac-proxy\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:41.683384 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.683160    2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-web-config\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:41.683384 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.683204    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vlzb\" (UniqueName: \"kubernetes.io/projected/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-kube-api-access-8vlzb\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:41.683384 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.683217    2569 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-grpc-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:41.683384 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.683229    2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-config-out\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:41.683384 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.683241    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:41.683384 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.683255    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:41.683384 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.683266    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:41.683384 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.683289    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-prometheus-k8s-db\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:41.683384 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.683297    2569 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-secret-metrics-client-certs\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:41.683384 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:41.683307    2569 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4-config\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:08:42.418107 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.418069    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4","Type":"ContainerDied","Data":"a794c28985528d2e21702bc8c898c9944a854ca6e1f1b10a2b70f2477ad9269f"}
May 25 13:08:42.418553 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.418117    2569 scope.go:117] "RemoveContainer" containerID="e1a2c9f12a192a69504fd7b7e2140f87078af39c219ebe78a6efedd28b27526c"
May 25 13:08:42.418553 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.418142    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.426646 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.426251    2569 scope.go:117] "RemoveContainer" containerID="96b6458cd0c6dde2dc4cda0c284fcbb80c327d882f90c8bf3cf19e9a124a588d"
May 25 13:08:42.435552 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.435530    2569 scope.go:117] "RemoveContainer" containerID="bf3c81701e53026ca856ca50846a11fcb5aa23b78110c7e404d30dd881140bd2"
May 25 13:08:42.442684 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.442661    2569 scope.go:117] "RemoveContainer" containerID="8467305e0f33c532e4bca8ea94cdffbbf30f129d0d219237673a2e4e64538c53"
May 25 13:08:42.446390 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.446347    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 13:08:42.450910 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.450890    2569 scope.go:117] "RemoveContainer" containerID="5408f6a405b115e05f8413691be2621edf156bc52cf81b0ec71e52670c5f68fe"
May 25 13:08:42.451354 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.451334    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 13:08:42.457955 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.457931    2569 scope.go:117] "RemoveContainer" containerID="161784d54901e7c3d063ce6c39404cab72424acdf62c96035ae41865c7763ccd"
May 25 13:08:42.465548 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.465525    2569 scope.go:117] "RemoveContainer" containerID="915c5c6308a52cb75efa1900a479c023ad49bb4e1b3aea398d9665efd4d0360e"
May 25 13:08:42.476840 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.476815    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 13:08:42.477141 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477128    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b30c873-1845-484c-ae06-b3eab38d772d" containerName="registry"
May 25 13:08:42.477196 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477142    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b30c873-1845-484c-ae06-b3eab38d772d" containerName="registry"
May 25 13:08:42.477196 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477151    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="thanos-sidecar"
May 25 13:08:42.477196 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477156    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="thanos-sidecar"
May 25 13:08:42.477196 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477181    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="prometheus"
May 25 13:08:42.477196 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477189    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="prometheus"
May 25 13:08:42.477196 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477196    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="kube-rbac-proxy"
May 25 13:08:42.477370 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477201    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="kube-rbac-proxy"
May 25 13:08:42.477370 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477210    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="config-reloader"
May 25 13:08:42.477370 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477214    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="config-reloader"
May 25 13:08:42.477370 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477222    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="kube-rbac-proxy-thanos"
May 25 13:08:42.477370 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477229    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="kube-rbac-proxy-thanos"
May 25 13:08:42.477370 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477236    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="init-config-reloader"
May 25 13:08:42.477370 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477241    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="init-config-reloader"
May 25 13:08:42.477370 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477256    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="kube-rbac-proxy-web"
May 25 13:08:42.477370 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477261    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="kube-rbac-proxy-web"
May 25 13:08:42.477370 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477267    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bff5617f-f53f-4d47-afb9-4b68d3b159ed" containerName="console"
May 25 13:08:42.477370 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477272    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bff5617f-f53f-4d47-afb9-4b68d3b159ed" containerName="console"
May 25 13:08:42.477370 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477319    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="kube-rbac-proxy-web"
May 25 13:08:42.477370 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477327    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="kube-rbac-proxy"
May 25 13:08:42.477370 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477334    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="kube-rbac-proxy-thanos"
May 25 13:08:42.477370 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477343    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="prometheus"
May 25 13:08:42.477370 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477349    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="thanos-sidecar"
May 25 13:08:42.477370 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477355    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="bff5617f-f53f-4d47-afb9-4b68d3b159ed" containerName="console"
May 25 13:08:42.477370 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477362    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" containerName="config-reloader"
May 25 13:08:42.477370 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.477369    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="4b30c873-1845-484c-ae06-b3eab38d772d" containerName="registry"
May 25 13:08:42.482638 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.482618    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.487036 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.487013    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 25 13:08:42.487528 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.487508    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 25 13:08:42.487623 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.487526    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 25 13:08:42.487623 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.487531    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 25 13:08:42.487623 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.487516    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 25 13:08:42.487953 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.487933    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-947586grgijv7\""
May 25 13:08:42.488048 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.488000    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hv275\""
May 25 13:08:42.488232 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.488218    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 25 13:08:42.488690 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.488582    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 25 13:08:42.488889 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.488872    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 25 13:08:42.488994 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.488974    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 25 13:08:42.489083 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.489067    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 25 13:08:42.489154 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.489067    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 25 13:08:42.497091 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.497069    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 25 13:08:42.500431 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.500413    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 25 13:08:42.506203 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.506157    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 13:08:42.590709 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.590611    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7d8f3fe-873f-46ff-a897-4912c1434fd7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.590709 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.590654    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.590709 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.590688    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c7d8f3fe-873f-46ff-a897-4912c1434fd7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.590947 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.590721    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.590947 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.590759    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7d8f3fe-873f-46ff-a897-4912c1434fd7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.590947 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.590776    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7d8f3fe-873f-46ff-a897-4912c1434fd7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.590947 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.590791    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-web-config\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.590947 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.590807    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.590947 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.590823    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tc9r\" (UniqueName: \"kubernetes.io/projected/c7d8f3fe-873f-46ff-a897-4912c1434fd7-kube-api-access-6tc9r\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.590947 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.590873    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7d8f3fe-873f-46ff-a897-4912c1434fd7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.590947 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.590904    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.590947 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.590921    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.590947 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.590938    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c7d8f3fe-873f-46ff-a897-4912c1434fd7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.591307 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.590952    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c7d8f3fe-873f-46ff-a897-4912c1434fd7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.591307 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.590989    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.591307 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.591061    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.591307 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.591093    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c7d8f3fe-873f-46ff-a897-4912c1434fd7-config-out\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.591307 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.591116    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-config\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.596614 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.596581    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4" path="/var/lib/kubelet/pods/9ea0ba4d-57bd-4c30-9e4f-7e356a3824c4/volumes"
May 25 13:08:42.692237 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.692204    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6tc9r\" (UniqueName: \"kubernetes.io/projected/c7d8f3fe-873f-46ff-a897-4912c1434fd7-kube-api-access-6tc9r\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.692413 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.692242    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7d8f3fe-873f-46ff-a897-4912c1434fd7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.692413 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.692284    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.692413 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.692310    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.692413 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.692338    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c7d8f3fe-873f-46ff-a897-4912c1434fd7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.692413 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.692365    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c7d8f3fe-873f-46ff-a897-4912c1434fd7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.692712 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.692525    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.692712 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.692569    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.692712 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.692622    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c7d8f3fe-873f-46ff-a897-4912c1434fd7-config-out\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.692712 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.692674    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-config\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.692913 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.692718    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7d8f3fe-873f-46ff-a897-4912c1434fd7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.692913 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.692751    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.692913 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.692752    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c7d8f3fe-873f-46ff-a897-4912c1434fd7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.692913 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.692779    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c7d8f3fe-873f-46ff-a897-4912c1434fd7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.692913 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.692805    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.692913 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.692842    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7d8f3fe-873f-46ff-a897-4912c1434fd7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.692913 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.692871    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7d8f3fe-873f-46ff-a897-4912c1434fd7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.692913 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.692897    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-web-config\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.693310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.692942    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.693310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.693193    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7d8f3fe-873f-46ff-a897-4912c1434fd7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.695728 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.695586    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.695728 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.695676    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.695728 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.695722    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.695974 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.695886    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-config\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.696034 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.695992    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.696891 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.696243    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.696891 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.696534    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c7d8f3fe-873f-46ff-a897-4912c1434fd7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.696891 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.696770    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7d8f3fe-873f-46ff-a897-4912c1434fd7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.696891 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.696845    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.696891 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.696852    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7d8f3fe-873f-46ff-a897-4912c1434fd7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.697215 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.697156    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7d8f3fe-873f-46ff-a897-4912c1434fd7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.697929 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.697902    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.698360 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.698339    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c7d8f3fe-873f-46ff-a897-4912c1434fd7-web-config\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.698446 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.698425    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c7d8f3fe-873f-46ff-a897-4912c1434fd7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.698630 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.698611    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c7d8f3fe-873f-46ff-a897-4912c1434fd7-config-out\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.699957 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.699939    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tc9r\" (UniqueName: \"kubernetes.io/projected/c7d8f3fe-873f-46ff-a897-4912c1434fd7-kube-api-access-6tc9r\") pod \"prometheus-k8s-0\" (UID: \"c7d8f3fe-873f-46ff-a897-4912c1434fd7\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.792940 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.792897    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:42.932376 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:42.932349    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 13:08:42.934754 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:08:42.934728    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d8f3fe_873f_46ff_a897_4912c1434fd7.slice/crio-8e233682429a0594d06ac47adad82d3bc5d6855a39ec3dea002a84d8842f1cd5 WatchSource:0}: Error finding container 8e233682429a0594d06ac47adad82d3bc5d6855a39ec3dea002a84d8842f1cd5: Status 404 returned error can't find the container with id 8e233682429a0594d06ac47adad82d3bc5d6855a39ec3dea002a84d8842f1cd5
May 25 13:08:43.422298 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:43.422262    2569 generic.go:358] "Generic (PLEG): container finished" podID="c7d8f3fe-873f-46ff-a897-4912c1434fd7" containerID="7b19a383db82d7cc277713565c86ae97d40888fea62db1a64f6f67860fba4ec4" exitCode=0
May 25 13:08:43.422753 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:43.422346    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c7d8f3fe-873f-46ff-a897-4912c1434fd7","Type":"ContainerDied","Data":"7b19a383db82d7cc277713565c86ae97d40888fea62db1a64f6f67860fba4ec4"}
May 25 13:08:43.422753 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:43.422380    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c7d8f3fe-873f-46ff-a897-4912c1434fd7","Type":"ContainerStarted","Data":"8e233682429a0594d06ac47adad82d3bc5d6855a39ec3dea002a84d8842f1cd5"}
May 25 13:08:44.429994 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:44.429949    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c7d8f3fe-873f-46ff-a897-4912c1434fd7","Type":"ContainerStarted","Data":"d48ecf8bca573c20e57e5571bf4c0b9cc9a698c9df39e03f2664d9789b788952"}
May 25 13:08:44.429994 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:44.429994    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c7d8f3fe-873f-46ff-a897-4912c1434fd7","Type":"ContainerStarted","Data":"e1c4d6cf9cbab21b6c520e3e23256edb8b8cb0211b9d2c1ad9f2f0d01af171b2"}
May 25 13:08:44.430431 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:44.430007    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c7d8f3fe-873f-46ff-a897-4912c1434fd7","Type":"ContainerStarted","Data":"73c64e9b3c9ae25c1f8130b1ae706ce889805fbc0b86237a807f5bf0a1d253fc"}
May 25 13:08:44.430431 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:44.430019    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c7d8f3fe-873f-46ff-a897-4912c1434fd7","Type":"ContainerStarted","Data":"c6083de0f2e7855968126845b4bcab35ec93083ac964bd159cb5754fc9781e27"}
May 25 13:08:44.430431 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:44.430032    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c7d8f3fe-873f-46ff-a897-4912c1434fd7","Type":"ContainerStarted","Data":"83654528b8d2a9b2d533f219a9f810b996316685c514424d521f2a3882a1fc79"}
May 25 13:08:44.430431 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:44.430044    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c7d8f3fe-873f-46ff-a897-4912c1434fd7","Type":"ContainerStarted","Data":"f81351bcedda3208cc9e65b6d0b28001910f60870497e498f2034ef49ed3af72"}
May 25 13:08:44.458198 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:44.458133    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.4581184289999998 podStartE2EDuration="2.458118429s" podCreationTimestamp="2026-05-25 13:08:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:08:44.456031504 +0000 UTC m=+240.458773301" watchObservedRunningTime="2026-05-25 13:08:44.458118429 +0000 UTC m=+240.460860183"
May 25 13:08:47.794013 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:47.793974    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:08:56.518789 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:56.518731    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs\") pod \"network-metrics-daemon-8rxxc\" (UID: \"8878a1b9-7848-4f10-a0f6-f88059209e87\") " pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:08:56.521051 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:56.521015    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8878a1b9-7848-4f10-a0f6-f88059209e87-metrics-certs\") pod \"network-metrics-daemon-8rxxc\" (UID: \"8878a1b9-7848-4f10-a0f6-f88059209e87\") " pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:08:56.700661 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:56.700627    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rx2c5\""
May 25 13:08:56.708714 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:56.708685    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rxxc"
May 25 13:08:56.837787 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:56.837751    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8rxxc"]
May 25 13:08:56.840987 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:08:56.840960    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8878a1b9_7848_4f10_a0f6_f88059209e87.slice/crio-e4e0fddd2a90535704b05ac86ce10b5d3108154823025695753a8a14d4ff4cb4 WatchSource:0}: Error finding container e4e0fddd2a90535704b05ac86ce10b5d3108154823025695753a8a14d4ff4cb4: Status 404 returned error can't find the container with id e4e0fddd2a90535704b05ac86ce10b5d3108154823025695753a8a14d4ff4cb4
May 25 13:08:57.475024 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:57.474973    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8rxxc" event={"ID":"8878a1b9-7848-4f10-a0f6-f88059209e87","Type":"ContainerStarted","Data":"e4e0fddd2a90535704b05ac86ce10b5d3108154823025695753a8a14d4ff4cb4"}
May 25 13:08:58.484940 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:58.484903    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8rxxc" event={"ID":"8878a1b9-7848-4f10-a0f6-f88059209e87","Type":"ContainerStarted","Data":"a10c7595b877162aeb891df0f9898d1bd66162ad0cae2dd1ae589e2aec66e8f1"}
May 25 13:08:58.484940 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:58.484941    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8rxxc" event={"ID":"8878a1b9-7848-4f10-a0f6-f88059209e87","Type":"ContainerStarted","Data":"e9b08b448f911ebf86b65dea887dd66ceed00d094b3528828c80e65ba69f807c"}
May 25 13:08:58.500621 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:08:58.500573    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8rxxc" podStartSLOduration=252.449690777 podStartE2EDuration="4m13.500558081s" podCreationTimestamp="2026-05-25 13:04:45 +0000 UTC" firstStartedPulling="2026-05-25 13:08:56.843373636 +0000 UTC m=+252.846115371" lastFinishedPulling="2026-05-25 13:08:57.89424094 +0000 UTC m=+253.896982675" observedRunningTime="2026-05-25 13:08:58.49949027 +0000 UTC m=+254.502232024" watchObservedRunningTime="2026-05-25 13:08:58.500558081 +0000 UTC m=+254.503299834"
May 25 13:09:42.794109 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:09:42.794067    2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:09:42.809532 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:09:42.809504    2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:09:43.638467 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:09:43.638437    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 25 13:09:44.524534 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:09:44.524501    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/1.log"
May 25 13:09:44.525037 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:09:44.524502    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/1.log"
May 25 13:09:44.530403 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:09:44.530382    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:09:44.530580 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:09:44.530484    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:09:44.534599 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:09:44.534578    2569 kubelet.go:1628] "Image garbage collection succeeded"
May 25 13:13:54.749144 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:13:54.749107    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-rgssw"]
May 25 13:13:54.752673 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:13:54.752652    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-rgssw"
May 25 13:13:54.755193 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:13:54.755144    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
May 25 13:13:54.755338 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:13:54.755190    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 25 13:13:54.755338 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:13:54.755253    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 25 13:13:54.756110 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:13:54.756090    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-h9v8f\""
May 25 13:13:54.764288 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:13:54.764258    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-rgssw"]
May 25 13:13:54.808236 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:13:54.808189    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvc78\" (UniqueName: \"kubernetes.io/projected/33488dd0-5667-45b0-8fba-5375bb11f1c7-kube-api-access-xvc78\") pod \"odh-model-controller-696fc77849-rgssw\" (UID: \"33488dd0-5667-45b0-8fba-5375bb11f1c7\") " pod="kserve/odh-model-controller-696fc77849-rgssw"
May 25 13:13:54.808418 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:13:54.808274    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33488dd0-5667-45b0-8fba-5375bb11f1c7-cert\") pod \"odh-model-controller-696fc77849-rgssw\" (UID: \"33488dd0-5667-45b0-8fba-5375bb11f1c7\") " pod="kserve/odh-model-controller-696fc77849-rgssw"
May 25 13:13:54.909057 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:13:54.909012    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xvc78\" (UniqueName: \"kubernetes.io/projected/33488dd0-5667-45b0-8fba-5375bb11f1c7-kube-api-access-xvc78\") pod \"odh-model-controller-696fc77849-rgssw\" (UID: \"33488dd0-5667-45b0-8fba-5375bb11f1c7\") " pod="kserve/odh-model-controller-696fc77849-rgssw"
May 25 13:13:54.909305 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:13:54.909095    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33488dd0-5667-45b0-8fba-5375bb11f1c7-cert\") pod \"odh-model-controller-696fc77849-rgssw\" (UID: \"33488dd0-5667-45b0-8fba-5375bb11f1c7\") " pod="kserve/odh-model-controller-696fc77849-rgssw"
May 25 13:13:54.911519 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:13:54.911488    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33488dd0-5667-45b0-8fba-5375bb11f1c7-cert\") pod \"odh-model-controller-696fc77849-rgssw\" (UID: \"33488dd0-5667-45b0-8fba-5375bb11f1c7\") " pod="kserve/odh-model-controller-696fc77849-rgssw"
May 25 13:13:54.920339 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:13:54.920303    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvc78\" (UniqueName: \"kubernetes.io/projected/33488dd0-5667-45b0-8fba-5375bb11f1c7-kube-api-access-xvc78\") pod \"odh-model-controller-696fc77849-rgssw\" (UID: \"33488dd0-5667-45b0-8fba-5375bb11f1c7\") " pod="kserve/odh-model-controller-696fc77849-rgssw"
May 25 13:13:55.063467 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:13:55.063367    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-rgssw"
May 25 13:13:55.194780 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:13:55.194746    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-rgssw"]
May 25 13:13:55.197779 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:13:55.197744    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33488dd0_5667_45b0_8fba_5375bb11f1c7.slice/crio-11cd03127ae1ed32833fd940c40067488d0f514c8bbd59ec98ab73d5ff59ed6f WatchSource:0}: Error finding container 11cd03127ae1ed32833fd940c40067488d0f514c8bbd59ec98ab73d5ff59ed6f: Status 404 returned error can't find the container with id 11cd03127ae1ed32833fd940c40067488d0f514c8bbd59ec98ab73d5ff59ed6f
May 25 13:13:55.198895 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:13:55.198877    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:13:55.392375 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:13:55.392338    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-rgssw" event={"ID":"33488dd0-5667-45b0-8fba-5375bb11f1c7","Type":"ContainerStarted","Data":"11cd03127ae1ed32833fd940c40067488d0f514c8bbd59ec98ab73d5ff59ed6f"}
May 25 13:13:58.403940 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:13:58.403845    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-rgssw" event={"ID":"33488dd0-5667-45b0-8fba-5375bb11f1c7","Type":"ContainerStarted","Data":"1a5a3f94ff834aadfa7d771c07b81aa2d9ee4f65922233a673c701f99f56fc92"}
May 25 13:13:58.404384 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:13:58.404013    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-rgssw"
May 25 13:13:58.420577 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:13:58.420519    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-rgssw" podStartSLOduration=1.505664582 podStartE2EDuration="4.420504661s" podCreationTimestamp="2026-05-25 13:13:54 +0000 UTC" firstStartedPulling="2026-05-25 13:13:55.198995964 +0000 UTC m=+551.201737701" lastFinishedPulling="2026-05-25 13:13:58.113836033 +0000 UTC m=+554.116577780" observedRunningTime="2026-05-25 13:13:58.41873474 +0000 UTC m=+554.421476489" watchObservedRunningTime="2026-05-25 13:13:58.420504661 +0000 UTC m=+554.423246414"
May 25 13:14:09.409034 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:09.408998    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-rgssw"
May 25 13:14:10.242273 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:10.242237    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-5pskx"]
May 25 13:14:10.245786 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:10.245766    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5pskx"
May 25 13:14:10.248449 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:10.248422    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-r99bh\""
May 25 13:14:10.248561 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:10.248427    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 25 13:14:10.253511 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:10.253486    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-5pskx"]
May 25 13:14:10.351287 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:10.351252    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-858q5\" (UniqueName: \"kubernetes.io/projected/dbab6580-d2ae-49e4-93c5-1d7c345f3e44-kube-api-access-858q5\") pod \"s3-init-5pskx\" (UID: \"dbab6580-d2ae-49e4-93c5-1d7c345f3e44\") " pod="kserve/s3-init-5pskx"
May 25 13:14:10.452237 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:10.452194    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-858q5\" (UniqueName: \"kubernetes.io/projected/dbab6580-d2ae-49e4-93c5-1d7c345f3e44-kube-api-access-858q5\") pod \"s3-init-5pskx\" (UID: \"dbab6580-d2ae-49e4-93c5-1d7c345f3e44\") " pod="kserve/s3-init-5pskx"
May 25 13:14:10.461407 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:10.461376    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-858q5\" (UniqueName: \"kubernetes.io/projected/dbab6580-d2ae-49e4-93c5-1d7c345f3e44-kube-api-access-858q5\") pod \"s3-init-5pskx\" (UID: \"dbab6580-d2ae-49e4-93c5-1d7c345f3e44\") " pod="kserve/s3-init-5pskx"
May 25 13:14:10.572577 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:10.572493    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5pskx"
May 25 13:14:10.701526 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:10.701498    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-5pskx"]
May 25 13:14:10.704064 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:14:10.704029    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbab6580_d2ae_49e4_93c5_1d7c345f3e44.slice/crio-395821dd899249affea1ce7275d7ba28d6112e9f5f40703041ac90d91d66901c WatchSource:0}: Error finding container 395821dd899249affea1ce7275d7ba28d6112e9f5f40703041ac90d91d66901c: Status 404 returned error can't find the container with id 395821dd899249affea1ce7275d7ba28d6112e9f5f40703041ac90d91d66901c
May 25 13:14:11.447343 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:11.447280    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5pskx" event={"ID":"dbab6580-d2ae-49e4-93c5-1d7c345f3e44","Type":"ContainerStarted","Data":"395821dd899249affea1ce7275d7ba28d6112e9f5f40703041ac90d91d66901c"}
May 25 13:14:16.464909 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:16.464865    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5pskx" event={"ID":"dbab6580-d2ae-49e4-93c5-1d7c345f3e44","Type":"ContainerStarted","Data":"1ecf7421090fa137d1acdae818f956c3b2a2574a9023543741a54445a894de22"}
May 25 13:14:16.481653 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:16.481590    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-5pskx" podStartSLOduration=1.674183986 podStartE2EDuration="6.481575197s" podCreationTimestamp="2026-05-25 13:14:10 +0000 UTC" firstStartedPulling="2026-05-25 13:14:10.705876879 +0000 UTC m=+566.708618616" lastFinishedPulling="2026-05-25 13:14:15.513268095 +0000 UTC m=+571.516009827" observedRunningTime="2026-05-25 13:14:16.480823508 +0000 UTC m=+572.483565261" watchObservedRunningTime="2026-05-25 13:14:16.481575197 +0000 UTC m=+572.484316951"
May 25 13:14:19.475233 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:19.475151    2569 generic.go:358] "Generic (PLEG): container finished" podID="dbab6580-d2ae-49e4-93c5-1d7c345f3e44" containerID="1ecf7421090fa137d1acdae818f956c3b2a2574a9023543741a54445a894de22" exitCode=0
May 25 13:14:19.475233 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:19.475221    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5pskx" event={"ID":"dbab6580-d2ae-49e4-93c5-1d7c345f3e44","Type":"ContainerDied","Data":"1ecf7421090fa137d1acdae818f956c3b2a2574a9023543741a54445a894de22"}
May 25 13:14:20.599552 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:20.599521    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5pskx"
May 25 13:14:20.742312 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:20.742213    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-858q5\" (UniqueName: \"kubernetes.io/projected/dbab6580-d2ae-49e4-93c5-1d7c345f3e44-kube-api-access-858q5\") pod \"dbab6580-d2ae-49e4-93c5-1d7c345f3e44\" (UID: \"dbab6580-d2ae-49e4-93c5-1d7c345f3e44\") "
May 25 13:14:20.744388 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:20.744358    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbab6580-d2ae-49e4-93c5-1d7c345f3e44-kube-api-access-858q5" (OuterVolumeSpecName: "kube-api-access-858q5") pod "dbab6580-d2ae-49e4-93c5-1d7c345f3e44" (UID: "dbab6580-d2ae-49e4-93c5-1d7c345f3e44"). InnerVolumeSpecName "kube-api-access-858q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:14:20.842944 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:20.842912    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-858q5\" (UniqueName: \"kubernetes.io/projected/dbab6580-d2ae-49e4-93c5-1d7c345f3e44-kube-api-access-858q5\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:14:21.483038 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:21.483004    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5pskx" event={"ID":"dbab6580-d2ae-49e4-93c5-1d7c345f3e44","Type":"ContainerDied","Data":"395821dd899249affea1ce7275d7ba28d6112e9f5f40703041ac90d91d66901c"}
May 25 13:14:21.483038 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:21.483038    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="395821dd899249affea1ce7275d7ba28d6112e9f5f40703041ac90d91d66901c"
May 25 13:14:21.483038 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:21.483041    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5pskx"
May 25 13:14:31.044865 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:31.044827    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"]
May 25 13:14:31.045360 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:31.045162    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbab6580-d2ae-49e4-93c5-1d7c345f3e44" containerName="s3-init"
May 25 13:14:31.045360 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:31.045197    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbab6580-d2ae-49e4-93c5-1d7c345f3e44" containerName="s3-init"
May 25 13:14:31.045360 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:31.045256    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="dbab6580-d2ae-49e4-93c5-1d7c345f3e44" containerName="s3-init"
May 25 13:14:31.048489 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:31.048466    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"
May 25 13:14:31.051006 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:31.050977    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-889c1-kube-rbac-proxy-sar-config\""
May 25 13:14:31.051317 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:31.051300    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-889c1-predictor-serving-cert\""
May 25 13:14:31.051543 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:31.051338    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 13:14:31.051543 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:31.051362    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7pmnd\""
May 25 13:14:31.051971 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:31.051947    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 13:14:31.058914 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:31.058889    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"]
May 25 13:14:31.238380 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:31.238348    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrzvs\" (UniqueName: \"kubernetes.io/projected/fe91929d-2d86-4a10-9dc9-8ffb2a26434f-kube-api-access-vrzvs\") pod \"error-404-isvc-889c1-predictor-b649dc755-pwsvf\" (UID: \"fe91929d-2d86-4a10-9dc9-8ffb2a26434f\") " pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"
May 25 13:14:31.238569 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:31.238396    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-889c1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe91929d-2d86-4a10-9dc9-8ffb2a26434f-error-404-isvc-889c1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-889c1-predictor-b649dc755-pwsvf\" (UID: \"fe91929d-2d86-4a10-9dc9-8ffb2a26434f\") " pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"
May 25 13:14:31.238569 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:31.238503    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe91929d-2d86-4a10-9dc9-8ffb2a26434f-proxy-tls\") pod \"error-404-isvc-889c1-predictor-b649dc755-pwsvf\" (UID: \"fe91929d-2d86-4a10-9dc9-8ffb2a26434f\") " pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"
May 25 13:14:31.339914 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:31.339819    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe91929d-2d86-4a10-9dc9-8ffb2a26434f-proxy-tls\") pod \"error-404-isvc-889c1-predictor-b649dc755-pwsvf\" (UID: \"fe91929d-2d86-4a10-9dc9-8ffb2a26434f\") " pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"
May 25 13:14:31.340088 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:31.339926    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vrzvs\" (UniqueName: \"kubernetes.io/projected/fe91929d-2d86-4a10-9dc9-8ffb2a26434f-kube-api-access-vrzvs\") pod \"error-404-isvc-889c1-predictor-b649dc755-pwsvf\" (UID: \"fe91929d-2d86-4a10-9dc9-8ffb2a26434f\") " pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"
May 25 13:14:31.340088 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:14:31.339971    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-889c1-predictor-serving-cert: secret "error-404-isvc-889c1-predictor-serving-cert" not found
May 25 13:14:31.340088 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:31.340007    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-889c1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe91929d-2d86-4a10-9dc9-8ffb2a26434f-error-404-isvc-889c1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-889c1-predictor-b649dc755-pwsvf\" (UID: \"fe91929d-2d86-4a10-9dc9-8ffb2a26434f\") " pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"
May 25 13:14:31.340088 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:14:31.340061    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe91929d-2d86-4a10-9dc9-8ffb2a26434f-proxy-tls podName:fe91929d-2d86-4a10-9dc9-8ffb2a26434f nodeName:}" failed. No retries permitted until 2026-05-25 13:14:31.840040935 +0000 UTC m=+587.842782667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fe91929d-2d86-4a10-9dc9-8ffb2a26434f-proxy-tls") pod "error-404-isvc-889c1-predictor-b649dc755-pwsvf" (UID: "fe91929d-2d86-4a10-9dc9-8ffb2a26434f") : secret "error-404-isvc-889c1-predictor-serving-cert" not found
May 25 13:14:31.340701 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:31.340678    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-889c1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe91929d-2d86-4a10-9dc9-8ffb2a26434f-error-404-isvc-889c1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-889c1-predictor-b649dc755-pwsvf\" (UID: \"fe91929d-2d86-4a10-9dc9-8ffb2a26434f\") " pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"
May 25 13:14:31.349405 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:31.349375    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrzvs\" (UniqueName: \"kubernetes.io/projected/fe91929d-2d86-4a10-9dc9-8ffb2a26434f-kube-api-access-vrzvs\") pod \"error-404-isvc-889c1-predictor-b649dc755-pwsvf\" (UID: \"fe91929d-2d86-4a10-9dc9-8ffb2a26434f\") " pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"
May 25 13:14:31.845431 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:31.845392    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe91929d-2d86-4a10-9dc9-8ffb2a26434f-proxy-tls\") pod \"error-404-isvc-889c1-predictor-b649dc755-pwsvf\" (UID: \"fe91929d-2d86-4a10-9dc9-8ffb2a26434f\") " pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"
May 25 13:14:31.847795 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:31.847768    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe91929d-2d86-4a10-9dc9-8ffb2a26434f-proxy-tls\") pod \"error-404-isvc-889c1-predictor-b649dc755-pwsvf\" (UID: \"fe91929d-2d86-4a10-9dc9-8ffb2a26434f\") " pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"
May 25 13:14:31.961717 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:31.961620    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"
May 25 13:14:32.096854 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:32.096818    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"]
May 25 13:14:32.099544 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:14:32.099516    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe91929d_2d86_4a10_9dc9_8ffb2a26434f.slice/crio-8136a50893f4a95b90d6332f4dd9803ecbc9aa44880163aa1de60788f311417c WatchSource:0}: Error finding container 8136a50893f4a95b90d6332f4dd9803ecbc9aa44880163aa1de60788f311417c: Status 404 returned error can't find the container with id 8136a50893f4a95b90d6332f4dd9803ecbc9aa44880163aa1de60788f311417c
May 25 13:14:32.518411 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:32.518376    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf" event={"ID":"fe91929d-2d86-4a10-9dc9-8ffb2a26434f","Type":"ContainerStarted","Data":"8136a50893f4a95b90d6332f4dd9803ecbc9aa44880163aa1de60788f311417c"}
May 25 13:14:43.556096 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:43.556010    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf" event={"ID":"fe91929d-2d86-4a10-9dc9-8ffb2a26434f","Type":"ContainerStarted","Data":"ec5c4d06ad58a7c1a50dce837ed47d8cddbb991f64ea714c0c589e3a88645db9"}
May 25 13:14:45.545895 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:45.545859    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/1.log"
May 25 13:14:45.546350 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:45.545909    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/1.log"
May 25 13:14:45.551252 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:45.551232    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:14:45.551396 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:45.551346    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:14:46.570550 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:46.570517    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf" event={"ID":"fe91929d-2d86-4a10-9dc9-8ffb2a26434f","Type":"ContainerStarted","Data":"35e48046104a2c899c9b77a57e9c4eea558c4370f38446b6ea71f34da23754f5"}
May 25 13:14:46.571064 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:46.570688    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"
May 25 13:14:46.590132 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:46.590087    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf" podStartSLOduration=2.110072123 podStartE2EDuration="15.590072859s" podCreationTimestamp="2026-05-25 13:14:31 +0000 UTC" firstStartedPulling="2026-05-25 13:14:32.101128439 +0000 UTC m=+588.103870171" lastFinishedPulling="2026-05-25 13:14:45.581129168 +0000 UTC m=+601.583870907" observedRunningTime="2026-05-25 13:14:46.587795957 +0000 UTC m=+602.590537721" watchObservedRunningTime="2026-05-25 13:14:46.590072859 +0000 UTC m=+602.592814615"
May 25 13:14:47.573614 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:47.573579    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"
May 25 13:14:47.574904 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:47.574874    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf" podUID="fe91929d-2d86-4a10-9dc9-8ffb2a26434f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
May 25 13:14:48.576221 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:48.576158    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf" podUID="fe91929d-2d86-4a10-9dc9-8ffb2a26434f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
May 25 13:14:53.580942 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:53.580914    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"
May 25 13:14:53.581559 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:14:53.581522    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf" podUID="fe91929d-2d86-4a10-9dc9-8ffb2a26434f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
May 25 13:15:03.582065 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:15:03.581976    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf" podUID="fe91929d-2d86-4a10-9dc9-8ffb2a26434f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
May 25 13:15:13.582191 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:15:13.582132    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf" podUID="fe91929d-2d86-4a10-9dc9-8ffb2a26434f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
May 25 13:15:23.581845 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:15:23.581806    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf" podUID="fe91929d-2d86-4a10-9dc9-8ffb2a26434f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
May 25 13:15:33.582372 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:15:33.582332    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"
May 25 13:15:50.679694 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:15:50.679655    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts"]
May 25 13:15:50.683079 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:15:50.683060    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts"
May 25 13:15:50.685366 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:15:50.685345    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-889c1-kube-rbac-proxy-sar-config\""
May 25 13:15:50.685526 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:15:50.685510    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-889c1-serving-cert\""
May 25 13:15:50.692249 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:15:50.692226    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts"]
May 25 13:15:50.835191 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:15:50.835132    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c58bfbf8-1df7-4757-9a67-89b8fb7745df-proxy-tls\") pod \"switch-graph-889c1-858b68998c-h8jts\" (UID: \"c58bfbf8-1df7-4757-9a67-89b8fb7745df\") " pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts"
May 25 13:15:50.835369 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:15:50.835249    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c58bfbf8-1df7-4757-9a67-89b8fb7745df-openshift-service-ca-bundle\") pod \"switch-graph-889c1-858b68998c-h8jts\" (UID: \"c58bfbf8-1df7-4757-9a67-89b8fb7745df\") " pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts"
May 25 13:15:50.936347 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:15:50.936243    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c58bfbf8-1df7-4757-9a67-89b8fb7745df-openshift-service-ca-bundle\") pod \"switch-graph-889c1-858b68998c-h8jts\" (UID: \"c58bfbf8-1df7-4757-9a67-89b8fb7745df\") " pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts"
May 25 13:15:50.936347 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:15:50.936303    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c58bfbf8-1df7-4757-9a67-89b8fb7745df-proxy-tls\") pod \"switch-graph-889c1-858b68998c-h8jts\" (UID: \"c58bfbf8-1df7-4757-9a67-89b8fb7745df\") " pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts"
May 25 13:15:50.936879 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:15:50.936855    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c58bfbf8-1df7-4757-9a67-89b8fb7745df-openshift-service-ca-bundle\") pod \"switch-graph-889c1-858b68998c-h8jts\" (UID: \"c58bfbf8-1df7-4757-9a67-89b8fb7745df\") " pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts"
May 25 13:15:50.938751 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:15:50.938729    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c58bfbf8-1df7-4757-9a67-89b8fb7745df-proxy-tls\") pod \"switch-graph-889c1-858b68998c-h8jts\" (UID: \"c58bfbf8-1df7-4757-9a67-89b8fb7745df\") " pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts"
May 25 13:15:50.995031 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:15:50.995000    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts"
May 25 13:15:51.121520 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:15:51.121494    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts"]
May 25 13:15:51.123984 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:15:51.123961    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58bfbf8_1df7_4757_9a67_89b8fb7745df.slice/crio-d137c32acd63b8c7a4fee5fed3d64e68281c0420b9c6a9d688ebb64da826f3b5 WatchSource:0}: Error finding container d137c32acd63b8c7a4fee5fed3d64e68281c0420b9c6a9d688ebb64da826f3b5: Status 404 returned error can't find the container with id d137c32acd63b8c7a4fee5fed3d64e68281c0420b9c6a9d688ebb64da826f3b5
May 25 13:15:51.769074 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:15:51.769031    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts" event={"ID":"c58bfbf8-1df7-4757-9a67-89b8fb7745df","Type":"ContainerStarted","Data":"d137c32acd63b8c7a4fee5fed3d64e68281c0420b9c6a9d688ebb64da826f3b5"}
May 25 13:15:53.779117 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:15:53.779022    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts" event={"ID":"c58bfbf8-1df7-4757-9a67-89b8fb7745df","Type":"ContainerStarted","Data":"338660e73a31cccd16405c28d15628acdad666e0156fdc4bb71822dba734548e"}
May 25 13:15:53.779516 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:15:53.779119    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts"
May 25 13:15:53.797711 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:15:53.797148    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts" podStartSLOduration=1.472861347 podStartE2EDuration="3.79712807s" podCreationTimestamp="2026-05-25 13:15:50 +0000 UTC" firstStartedPulling="2026-05-25 13:15:51.12576421 +0000 UTC m=+667.128505942" lastFinishedPulling="2026-05-25 13:15:53.450030934 +0000 UTC m=+669.452772665" observedRunningTime="2026-05-25 13:15:53.796135506 +0000 UTC m=+669.798877249" watchObservedRunningTime="2026-05-25 13:15:53.79712807 +0000 UTC m=+669.799869824"
May 25 13:15:59.788188 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:15:59.788136    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts"
May 25 13:16:00.805857 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:00.805823    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts"]
May 25 13:16:00.806270 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:00.806031    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts" podUID="c58bfbf8-1df7-4757-9a67-89b8fb7745df" containerName="switch-graph-889c1" containerID="cri-o://338660e73a31cccd16405c28d15628acdad666e0156fdc4bb71822dba734548e" gracePeriod=30
May 25 13:16:00.894609 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:00.894570    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"]
May 25 13:16:00.895003 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:00.894957    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf" podUID="fe91929d-2d86-4a10-9dc9-8ffb2a26434f" containerName="kserve-container" containerID="cri-o://ec5c4d06ad58a7c1a50dce837ed47d8cddbb991f64ea714c0c589e3a88645db9" gracePeriod=30
May 25 13:16:00.895133 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:00.895023    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf" podUID="fe91929d-2d86-4a10-9dc9-8ffb2a26434f" containerName="kube-rbac-proxy" containerID="cri-o://35e48046104a2c899c9b77a57e9c4eea558c4370f38446b6ea71f34da23754f5" gracePeriod=30
May 25 13:16:01.106955 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:01.106909    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"]
May 25 13:16:01.110582 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:01.110556    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"
May 25 13:16:01.113862 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:01.113827    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-339a4-kube-rbac-proxy-sar-config\""
May 25 13:16:01.113862 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:01.113835    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-339a4-predictor-serving-cert\""
May 25 13:16:01.123936 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:01.123909    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"]
May 25 13:16:01.230149 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:01.230111    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb-proxy-tls\") pod \"error-404-isvc-339a4-predictor-7cdc89ff49-ncgln\" (UID: \"ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb\") " pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"
May 25 13:16:01.230383 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:01.230271    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-339a4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb-error-404-isvc-339a4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-339a4-predictor-7cdc89ff49-ncgln\" (UID: \"ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb\") " pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"
May 25 13:16:01.230383 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:01.230321    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmdmx\" (UniqueName: \"kubernetes.io/projected/ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb-kube-api-access-xmdmx\") pod \"error-404-isvc-339a4-predictor-7cdc89ff49-ncgln\" (UID: \"ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb\") " pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"
May 25 13:16:01.331613 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:01.331569    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-339a4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb-error-404-isvc-339a4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-339a4-predictor-7cdc89ff49-ncgln\" (UID: \"ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb\") " pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"
May 25 13:16:01.331814 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:01.331621    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xmdmx\" (UniqueName: \"kubernetes.io/projected/ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb-kube-api-access-xmdmx\") pod \"error-404-isvc-339a4-predictor-7cdc89ff49-ncgln\" (UID: \"ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb\") " pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"
May 25 13:16:01.331814 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:01.331669    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb-proxy-tls\") pod \"error-404-isvc-339a4-predictor-7cdc89ff49-ncgln\" (UID: \"ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb\") " pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"
May 25 13:16:01.331929 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:16:01.331819    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-339a4-predictor-serving-cert: secret "error-404-isvc-339a4-predictor-serving-cert" not found
May 25 13:16:01.331929 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:16:01.331883    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb-proxy-tls podName:ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb nodeName:}" failed. No retries permitted until 2026-05-25 13:16:01.831864469 +0000 UTC m=+677.834606205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb-proxy-tls") pod "error-404-isvc-339a4-predictor-7cdc89ff49-ncgln" (UID: "ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb") : secret "error-404-isvc-339a4-predictor-serving-cert" not found
May 25 13:16:01.332298 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:01.332278    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-339a4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb-error-404-isvc-339a4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-339a4-predictor-7cdc89ff49-ncgln\" (UID: \"ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb\") " pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"
May 25 13:16:01.341687 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:01.341661    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmdmx\" (UniqueName: \"kubernetes.io/projected/ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb-kube-api-access-xmdmx\") pod \"error-404-isvc-339a4-predictor-7cdc89ff49-ncgln\" (UID: \"ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb\") " pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"
May 25 13:16:01.803010 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:01.802978    2569 generic.go:358] "Generic (PLEG): container finished" podID="fe91929d-2d86-4a10-9dc9-8ffb2a26434f" containerID="35e48046104a2c899c9b77a57e9c4eea558c4370f38446b6ea71f34da23754f5" exitCode=2
May 25 13:16:01.803213 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:01.803047    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf" event={"ID":"fe91929d-2d86-4a10-9dc9-8ffb2a26434f","Type":"ContainerDied","Data":"35e48046104a2c899c9b77a57e9c4eea558c4370f38446b6ea71f34da23754f5"}
May 25 13:16:01.836492 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:01.836451    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb-proxy-tls\") pod \"error-404-isvc-339a4-predictor-7cdc89ff49-ncgln\" (UID: \"ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb\") " pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"
May 25 13:16:01.838860 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:01.838837    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb-proxy-tls\") pod \"error-404-isvc-339a4-predictor-7cdc89ff49-ncgln\" (UID: \"ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb\") " pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"
May 25 13:16:02.020559 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:02.020528    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"
May 25 13:16:02.153879 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:02.153811    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"]
May 25 13:16:02.808375 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:02.808335    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln" event={"ID":"ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb","Type":"ContainerStarted","Data":"f4cef4a02c7ccc0abab1b91b7bb5724cd17527c0bcba98823a520716a243e18d"}
May 25 13:16:02.808375 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:02.808374    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln" event={"ID":"ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb","Type":"ContainerStarted","Data":"8544c3b2807283bedf8fca3e2d9c5eee08673476c898a9a01db2a56075f900b3"}
May 25 13:16:02.808580 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:02.808391    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln" event={"ID":"ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb","Type":"ContainerStarted","Data":"e31d233fb0313d932a7b06e7ef378371cea355c8322f441407a343db99bd2a58"}
May 25 13:16:02.808618 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:02.808593    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"
May 25 13:16:02.828674 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:02.828624    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln" podStartSLOduration=1.82860989 podStartE2EDuration="1.82860989s" podCreationTimestamp="2026-05-25 13:16:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:16:02.826030597 +0000 UTC m=+678.828772366" watchObservedRunningTime="2026-05-25 13:16:02.82860989 +0000 UTC m=+678.831351643"
May 25 13:16:03.576720 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:03.576671    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf" podUID="fe91929d-2d86-4a10-9dc9-8ffb2a26434f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
May 25 13:16:03.581637 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:03.581586    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf" podUID="fe91929d-2d86-4a10-9dc9-8ffb2a26434f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
May 25 13:16:03.812623 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:03.812579    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"
May 25 13:16:03.813785 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:03.813754    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln" podUID="ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 25 13:16:04.148453 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.148430    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"
May 25 13:16:04.259218 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.259102    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe91929d-2d86-4a10-9dc9-8ffb2a26434f-proxy-tls\") pod \"fe91929d-2d86-4a10-9dc9-8ffb2a26434f\" (UID: \"fe91929d-2d86-4a10-9dc9-8ffb2a26434f\") "
May 25 13:16:04.259218 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.259156    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrzvs\" (UniqueName: \"kubernetes.io/projected/fe91929d-2d86-4a10-9dc9-8ffb2a26434f-kube-api-access-vrzvs\") pod \"fe91929d-2d86-4a10-9dc9-8ffb2a26434f\" (UID: \"fe91929d-2d86-4a10-9dc9-8ffb2a26434f\") "
May 25 13:16:04.259391 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.259220    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-889c1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe91929d-2d86-4a10-9dc9-8ffb2a26434f-error-404-isvc-889c1-kube-rbac-proxy-sar-config\") pod \"fe91929d-2d86-4a10-9dc9-8ffb2a26434f\" (UID: \"fe91929d-2d86-4a10-9dc9-8ffb2a26434f\") "
May 25 13:16:04.259661 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.259621    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe91929d-2d86-4a10-9dc9-8ffb2a26434f-error-404-isvc-889c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-889c1-kube-rbac-proxy-sar-config") pod "fe91929d-2d86-4a10-9dc9-8ffb2a26434f" (UID: "fe91929d-2d86-4a10-9dc9-8ffb2a26434f"). InnerVolumeSpecName "error-404-isvc-889c1-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:16:04.261563 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.261518    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe91929d-2d86-4a10-9dc9-8ffb2a26434f-kube-api-access-vrzvs" (OuterVolumeSpecName: "kube-api-access-vrzvs") pod "fe91929d-2d86-4a10-9dc9-8ffb2a26434f" (UID: "fe91929d-2d86-4a10-9dc9-8ffb2a26434f"). InnerVolumeSpecName "kube-api-access-vrzvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:16:04.261691 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.261574    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe91929d-2d86-4a10-9dc9-8ffb2a26434f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fe91929d-2d86-4a10-9dc9-8ffb2a26434f" (UID: "fe91929d-2d86-4a10-9dc9-8ffb2a26434f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:16:04.360589 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.360551    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe91929d-2d86-4a10-9dc9-8ffb2a26434f-proxy-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:16:04.360589 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.360583    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrzvs\" (UniqueName: \"kubernetes.io/projected/fe91929d-2d86-4a10-9dc9-8ffb2a26434f-kube-api-access-vrzvs\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:16:04.360589 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.360595    2569 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-889c1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe91929d-2d86-4a10-9dc9-8ffb2a26434f-error-404-isvc-889c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:16:04.787374 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.787332    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts" podUID="c58bfbf8-1df7-4757-9a67-89b8fb7745df" containerName="switch-graph-889c1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:16:04.817326 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.817293    2569 generic.go:358] "Generic (PLEG): container finished" podID="fe91929d-2d86-4a10-9dc9-8ffb2a26434f" containerID="ec5c4d06ad58a7c1a50dce837ed47d8cddbb991f64ea714c0c589e3a88645db9" exitCode=0
May 25 13:16:04.817497 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.817363    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf" event={"ID":"fe91929d-2d86-4a10-9dc9-8ffb2a26434f","Type":"ContainerDied","Data":"ec5c4d06ad58a7c1a50dce837ed47d8cddbb991f64ea714c0c589e3a88645db9"}
May 25 13:16:04.817497 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.817404    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf" event={"ID":"fe91929d-2d86-4a10-9dc9-8ffb2a26434f","Type":"ContainerDied","Data":"8136a50893f4a95b90d6332f4dd9803ecbc9aa44880163aa1de60788f311417c"}
May 25 13:16:04.817497 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.817420    2569 scope.go:117] "RemoveContainer" containerID="35e48046104a2c899c9b77a57e9c4eea558c4370f38446b6ea71f34da23754f5"
May 25 13:16:04.817497 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.817470    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"
May 25 13:16:04.817749 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.817724    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln" podUID="ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 25 13:16:04.825561 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.825533    2569 scope.go:117] "RemoveContainer" containerID="ec5c4d06ad58a7c1a50dce837ed47d8cddbb991f64ea714c0c589e3a88645db9"
May 25 13:16:04.832745 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.832712    2569 scope.go:117] "RemoveContainer" containerID="35e48046104a2c899c9b77a57e9c4eea558c4370f38446b6ea71f34da23754f5"
May 25 13:16:04.833047 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:16:04.833018    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e48046104a2c899c9b77a57e9c4eea558c4370f38446b6ea71f34da23754f5\": container with ID starting with 35e48046104a2c899c9b77a57e9c4eea558c4370f38446b6ea71f34da23754f5 not found: ID does not exist" containerID="35e48046104a2c899c9b77a57e9c4eea558c4370f38446b6ea71f34da23754f5"
May 25 13:16:04.833124 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.833058    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e48046104a2c899c9b77a57e9c4eea558c4370f38446b6ea71f34da23754f5"} err="failed to get container status \"35e48046104a2c899c9b77a57e9c4eea558c4370f38446b6ea71f34da23754f5\": rpc error: code = NotFound desc = could not find container \"35e48046104a2c899c9b77a57e9c4eea558c4370f38446b6ea71f34da23754f5\": container with ID starting with 35e48046104a2c899c9b77a57e9c4eea558c4370f38446b6ea71f34da23754f5 not found: ID does not exist"
May 25 13:16:04.833124 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.833080    2569 scope.go:117] "RemoveContainer" containerID="ec5c4d06ad58a7c1a50dce837ed47d8cddbb991f64ea714c0c589e3a88645db9"
May 25 13:16:04.833405 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:16:04.833359    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec5c4d06ad58a7c1a50dce837ed47d8cddbb991f64ea714c0c589e3a88645db9\": container with ID starting with ec5c4d06ad58a7c1a50dce837ed47d8cddbb991f64ea714c0c589e3a88645db9 not found: ID does not exist" containerID="ec5c4d06ad58a7c1a50dce837ed47d8cddbb991f64ea714c0c589e3a88645db9"
May 25 13:16:04.833530 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.833394    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec5c4d06ad58a7c1a50dce837ed47d8cddbb991f64ea714c0c589e3a88645db9"} err="failed to get container status \"ec5c4d06ad58a7c1a50dce837ed47d8cddbb991f64ea714c0c589e3a88645db9\": rpc error: code = NotFound desc = could not find container \"ec5c4d06ad58a7c1a50dce837ed47d8cddbb991f64ea714c0c589e3a88645db9\": container with ID starting with ec5c4d06ad58a7c1a50dce837ed47d8cddbb991f64ea714c0c589e3a88645db9 not found: ID does not exist"
May 25 13:16:04.835242 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.835219    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"]
May 25 13:16:04.838964 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:04.838946    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-889c1-predictor-b649dc755-pwsvf"]
May 25 13:16:06.597457 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:06.597423    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe91929d-2d86-4a10-9dc9-8ffb2a26434f" path="/var/lib/kubelet/pods/fe91929d-2d86-4a10-9dc9-8ffb2a26434f/volumes"
May 25 13:16:09.787440 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:09.787385    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts" podUID="c58bfbf8-1df7-4757-9a67-89b8fb7745df" containerName="switch-graph-889c1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:16:09.821844 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:09.821815    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"
May 25 13:16:09.822361 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:09.822329    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln" podUID="ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 25 13:16:14.786564 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:14.786520    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts" podUID="c58bfbf8-1df7-4757-9a67-89b8fb7745df" containerName="switch-graph-889c1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:16:14.786987 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:14.786647    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts"
May 25 13:16:19.787131 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:19.787092    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts" podUID="c58bfbf8-1df7-4757-9a67-89b8fb7745df" containerName="switch-graph-889c1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:16:19.823052 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:19.823012    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln" podUID="ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 25 13:16:20.744559 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:20.744522    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz"]
May 25 13:16:20.745032 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:20.745019    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe91929d-2d86-4a10-9dc9-8ffb2a26434f" containerName="kserve-container"
May 25 13:16:20.745091 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:20.745035    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe91929d-2d86-4a10-9dc9-8ffb2a26434f" containerName="kserve-container"
May 25 13:16:20.745091 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:20.745046    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe91929d-2d86-4a10-9dc9-8ffb2a26434f" containerName="kube-rbac-proxy"
May 25 13:16:20.745091 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:20.745054    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe91929d-2d86-4a10-9dc9-8ffb2a26434f" containerName="kube-rbac-proxy"
May 25 13:16:20.745201 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:20.745154    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="fe91929d-2d86-4a10-9dc9-8ffb2a26434f" containerName="kserve-container"
May 25 13:16:20.745201 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:20.745195    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="fe91929d-2d86-4a10-9dc9-8ffb2a26434f" containerName="kube-rbac-proxy"
May 25 13:16:20.748764 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:20.748742    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz"
May 25 13:16:20.751364 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:20.751340    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
May 25 13:16:20.751479 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:20.751379    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
May 25 13:16:20.757254 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:20.757232    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz"]
May 25 13:16:20.798634 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:20.798595    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8fe0506-8e9e-4e56-9b22-950b88f4e6f8-proxy-tls\") pod \"model-chainer-866798f95b-9dhqz\" (UID: \"d8fe0506-8e9e-4e56-9b22-950b88f4e6f8\") " pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz"
May 25 13:16:20.799030 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:20.798699    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8fe0506-8e9e-4e56-9b22-950b88f4e6f8-openshift-service-ca-bundle\") pod \"model-chainer-866798f95b-9dhqz\" (UID: \"d8fe0506-8e9e-4e56-9b22-950b88f4e6f8\") " pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz"
May 25 13:16:20.899139 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:20.899093    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8fe0506-8e9e-4e56-9b22-950b88f4e6f8-proxy-tls\") pod \"model-chainer-866798f95b-9dhqz\" (UID: \"d8fe0506-8e9e-4e56-9b22-950b88f4e6f8\") " pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz"
May 25 13:16:20.899332 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:20.899159    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8fe0506-8e9e-4e56-9b22-950b88f4e6f8-openshift-service-ca-bundle\") pod \"model-chainer-866798f95b-9dhqz\" (UID: \"d8fe0506-8e9e-4e56-9b22-950b88f4e6f8\") " pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz"
May 25 13:16:20.899332 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:16:20.899280    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
May 25 13:16:20.899404 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:16:20.899352    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8fe0506-8e9e-4e56-9b22-950b88f4e6f8-proxy-tls podName:d8fe0506-8e9e-4e56-9b22-950b88f4e6f8 nodeName:}" failed. No retries permitted until 2026-05-25 13:16:21.399335311 +0000 UTC m=+697.402077042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d8fe0506-8e9e-4e56-9b22-950b88f4e6f8-proxy-tls") pod "model-chainer-866798f95b-9dhqz" (UID: "d8fe0506-8e9e-4e56-9b22-950b88f4e6f8") : secret "model-chainer-serving-cert" not found
May 25 13:16:20.899803 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:20.899784    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8fe0506-8e9e-4e56-9b22-950b88f4e6f8-openshift-service-ca-bundle\") pod \"model-chainer-866798f95b-9dhqz\" (UID: \"d8fe0506-8e9e-4e56-9b22-950b88f4e6f8\") " pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz"
May 25 13:16:21.403709 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:21.403661    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8fe0506-8e9e-4e56-9b22-950b88f4e6f8-proxy-tls\") pod \"model-chainer-866798f95b-9dhqz\" (UID: \"d8fe0506-8e9e-4e56-9b22-950b88f4e6f8\") " pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz"
May 25 13:16:21.406249 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:21.406226    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8fe0506-8e9e-4e56-9b22-950b88f4e6f8-proxy-tls\") pod \"model-chainer-866798f95b-9dhqz\" (UID: \"d8fe0506-8e9e-4e56-9b22-950b88f4e6f8\") " pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz"
May 25 13:16:21.660305 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:21.660216    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz"
May 25 13:16:21.792231 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:21.792194    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz"]
May 25 13:16:21.795123 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:16:21.795089    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8fe0506_8e9e_4e56_9b22_950b88f4e6f8.slice/crio-e555ce8ea6b30e34b4da719480357629252f66ae3020c17d90e2dc025a24269c WatchSource:0}: Error finding container e555ce8ea6b30e34b4da719480357629252f66ae3020c17d90e2dc025a24269c: Status 404 returned error can't find the container with id e555ce8ea6b30e34b4da719480357629252f66ae3020c17d90e2dc025a24269c
May 25 13:16:21.873250 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:21.873206    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz" event={"ID":"d8fe0506-8e9e-4e56-9b22-950b88f4e6f8","Type":"ContainerStarted","Data":"73c25c6532dc768daf5a7f7766503e02eef9922553698a801b5d0a25e50917ac"}
May 25 13:16:21.873725 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:21.873258    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz" event={"ID":"d8fe0506-8e9e-4e56-9b22-950b88f4e6f8","Type":"ContainerStarted","Data":"e555ce8ea6b30e34b4da719480357629252f66ae3020c17d90e2dc025a24269c"}
May 25 13:16:21.873725 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:21.873332    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz"
May 25 13:16:21.888966 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:21.888887    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz" podStartSLOduration=1.888871189 podStartE2EDuration="1.888871189s" podCreationTimestamp="2026-05-25 13:16:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:16:21.886911631 +0000 UTC m=+697.889653384" watchObservedRunningTime="2026-05-25 13:16:21.888871189 +0000 UTC m=+697.891612941"
May 25 13:16:24.786922 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:24.786887    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts" podUID="c58bfbf8-1df7-4757-9a67-89b8fb7745df" containerName="switch-graph-889c1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:16:27.882903 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:27.882868    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz"
May 25 13:16:29.786712 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:29.786674    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts" podUID="c58bfbf8-1df7-4757-9a67-89b8fb7745df" containerName="switch-graph-889c1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:16:29.822451 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:29.822403    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln" podUID="ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 25 13:16:30.816618 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:30.816584    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz"]
May 25 13:16:30.816986 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:30.816853    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz" podUID="d8fe0506-8e9e-4e56-9b22-950b88f4e6f8" containerName="model-chainer" containerID="cri-o://73c25c6532dc768daf5a7f7766503e02eef9922553698a801b5d0a25e50917ac" gracePeriod=30
May 25 13:16:30.904224 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:30.904190    2569 generic.go:358] "Generic (PLEG): container finished" podID="c58bfbf8-1df7-4757-9a67-89b8fb7745df" containerID="338660e73a31cccd16405c28d15628acdad666e0156fdc4bb71822dba734548e" exitCode=0
May 25 13:16:30.904385 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:30.904262    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts" event={"ID":"c58bfbf8-1df7-4757-9a67-89b8fb7745df","Type":"ContainerDied","Data":"338660e73a31cccd16405c28d15628acdad666e0156fdc4bb71822dba734548e"}
May 25 13:16:31.135147 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.135109    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"]
May 25 13:16:31.139061 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.139035    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"
May 25 13:16:31.142284 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.142260    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-08c99-predictor-serving-cert\""
May 25 13:16:31.142408 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.142286    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-08c99-kube-rbac-proxy-sar-config\""
May 25 13:16:31.148888 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.148861    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"]
May 25 13:16:31.191457 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.191423    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-08c99-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d968d149-a207-4af3-91bf-db1ae7d8b745-error-404-isvc-08c99-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-08c99-predictor-559bfc6b95-vpb62\" (UID: \"d968d149-a207-4af3-91bf-db1ae7d8b745\") " pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"
May 25 13:16:31.191457 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.191459    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdnnx\" (UniqueName: \"kubernetes.io/projected/d968d149-a207-4af3-91bf-db1ae7d8b745-kube-api-access-pdnnx\") pod \"error-404-isvc-08c99-predictor-559bfc6b95-vpb62\" (UID: \"d968d149-a207-4af3-91bf-db1ae7d8b745\") " pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"
May 25 13:16:31.191664 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.191572    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d968d149-a207-4af3-91bf-db1ae7d8b745-proxy-tls\") pod \"error-404-isvc-08c99-predictor-559bfc6b95-vpb62\" (UID: \"d968d149-a207-4af3-91bf-db1ae7d8b745\") " pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"
May 25 13:16:31.292738 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.292696    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d968d149-a207-4af3-91bf-db1ae7d8b745-proxy-tls\") pod \"error-404-isvc-08c99-predictor-559bfc6b95-vpb62\" (UID: \"d968d149-a207-4af3-91bf-db1ae7d8b745\") " pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"
May 25 13:16:31.292940 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.292754    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-08c99-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d968d149-a207-4af3-91bf-db1ae7d8b745-error-404-isvc-08c99-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-08c99-predictor-559bfc6b95-vpb62\" (UID: \"d968d149-a207-4af3-91bf-db1ae7d8b745\") " pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"
May 25 13:16:31.292940 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:16:31.292844    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-08c99-predictor-serving-cert: secret "error-404-isvc-08c99-predictor-serving-cert" not found
May 25 13:16:31.292940 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:16:31.292908    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d968d149-a207-4af3-91bf-db1ae7d8b745-proxy-tls podName:d968d149-a207-4af3-91bf-db1ae7d8b745 nodeName:}" failed. No retries permitted until 2026-05-25 13:16:31.792889602 +0000 UTC m=+707.795631334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d968d149-a207-4af3-91bf-db1ae7d8b745-proxy-tls") pod "error-404-isvc-08c99-predictor-559bfc6b95-vpb62" (UID: "d968d149-a207-4af3-91bf-db1ae7d8b745") : secret "error-404-isvc-08c99-predictor-serving-cert" not found
May 25 13:16:31.293110 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.292958    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pdnnx\" (UniqueName: \"kubernetes.io/projected/d968d149-a207-4af3-91bf-db1ae7d8b745-kube-api-access-pdnnx\") pod \"error-404-isvc-08c99-predictor-559bfc6b95-vpb62\" (UID: \"d968d149-a207-4af3-91bf-db1ae7d8b745\") " pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"
May 25 13:16:31.293406 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.293387    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-08c99-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d968d149-a207-4af3-91bf-db1ae7d8b745-error-404-isvc-08c99-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-08c99-predictor-559bfc6b95-vpb62\" (UID: \"d968d149-a207-4af3-91bf-db1ae7d8b745\") " pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"
May 25 13:16:31.302829 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.302800    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdnnx\" (UniqueName: \"kubernetes.io/projected/d968d149-a207-4af3-91bf-db1ae7d8b745-kube-api-access-pdnnx\") pod \"error-404-isvc-08c99-predictor-559bfc6b95-vpb62\" (UID: \"d968d149-a207-4af3-91bf-db1ae7d8b745\") " pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"
May 25 13:16:31.452524 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.452502    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts"
May 25 13:16:31.494418 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.494389    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c58bfbf8-1df7-4757-9a67-89b8fb7745df-openshift-service-ca-bundle\") pod \"c58bfbf8-1df7-4757-9a67-89b8fb7745df\" (UID: \"c58bfbf8-1df7-4757-9a67-89b8fb7745df\") "
May 25 13:16:31.494609 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.494461    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c58bfbf8-1df7-4757-9a67-89b8fb7745df-proxy-tls\") pod \"c58bfbf8-1df7-4757-9a67-89b8fb7745df\" (UID: \"c58bfbf8-1df7-4757-9a67-89b8fb7745df\") "
May 25 13:16:31.494736 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.494713    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c58bfbf8-1df7-4757-9a67-89b8fb7745df-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c58bfbf8-1df7-4757-9a67-89b8fb7745df" (UID: "c58bfbf8-1df7-4757-9a67-89b8fb7745df"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:16:31.496502 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.496478    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c58bfbf8-1df7-4757-9a67-89b8fb7745df-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c58bfbf8-1df7-4757-9a67-89b8fb7745df" (UID: "c58bfbf8-1df7-4757-9a67-89b8fb7745df"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:16:31.595311 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.595276    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c58bfbf8-1df7-4757-9a67-89b8fb7745df-openshift-service-ca-bundle\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:16:31.595311 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.595305    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c58bfbf8-1df7-4757-9a67-89b8fb7745df-proxy-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:16:31.797508 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.797408    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d968d149-a207-4af3-91bf-db1ae7d8b745-proxy-tls\") pod \"error-404-isvc-08c99-predictor-559bfc6b95-vpb62\" (UID: \"d968d149-a207-4af3-91bf-db1ae7d8b745\") " pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"
May 25 13:16:31.799887 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.799863    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d968d149-a207-4af3-91bf-db1ae7d8b745-proxy-tls\") pod \"error-404-isvc-08c99-predictor-559bfc6b95-vpb62\" (UID: \"d968d149-a207-4af3-91bf-db1ae7d8b745\") " pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"
May 25 13:16:31.909015 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.908970    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts" event={"ID":"c58bfbf8-1df7-4757-9a67-89b8fb7745df","Type":"ContainerDied","Data":"d137c32acd63b8c7a4fee5fed3d64e68281c0420b9c6a9d688ebb64da826f3b5"}
May 25 13:16:31.909015 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.909022    2569 scope.go:117] "RemoveContainer" containerID="338660e73a31cccd16405c28d15628acdad666e0156fdc4bb71822dba734548e"
May 25 13:16:31.909486 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.909033    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts"
May 25 13:16:31.931362 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.931326    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts"]
May 25 13:16:31.935110 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:31.935086    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-889c1-858b68998c-h8jts"]
May 25 13:16:32.049938 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:32.049853    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"
May 25 13:16:32.181851 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:32.181741    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"]
May 25 13:16:32.184757 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:16:32.184726    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd968d149_a207_4af3_91bf_db1ae7d8b745.slice/crio-96451f3a8e9a7728e48e5023d2cf75b06b07fb548d471f640238daa39d927911 WatchSource:0}: Error finding container 96451f3a8e9a7728e48e5023d2cf75b06b07fb548d471f640238daa39d927911: Status 404 returned error can't find the container with id 96451f3a8e9a7728e48e5023d2cf75b06b07fb548d471f640238daa39d927911
May 25 13:16:32.597276 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:32.597241    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c58bfbf8-1df7-4757-9a67-89b8fb7745df" path="/var/lib/kubelet/pods/c58bfbf8-1df7-4757-9a67-89b8fb7745df/volumes"
May 25 13:16:32.881779 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:32.881691    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz" podUID="d8fe0506-8e9e-4e56-9b22-950b88f4e6f8" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:16:32.914531 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:32.914493    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62" event={"ID":"d968d149-a207-4af3-91bf-db1ae7d8b745","Type":"ContainerStarted","Data":"6b565544c019ce6e9f75b2aca9c3e168e47595e8965f72eddda905d2b8e8014d"}
May 25 13:16:32.914907 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:32.914535    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62" event={"ID":"d968d149-a207-4af3-91bf-db1ae7d8b745","Type":"ContainerStarted","Data":"231818c82ca9a23c34f94fcc9e8f2068d8322f3d8dd0620000edd9bb7c9fe243"}
May 25 13:16:32.914907 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:32.914549    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62" event={"ID":"d968d149-a207-4af3-91bf-db1ae7d8b745","Type":"ContainerStarted","Data":"96451f3a8e9a7728e48e5023d2cf75b06b07fb548d471f640238daa39d927911"}
May 25 13:16:32.914907 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:32.914719    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"
May 25 13:16:32.932397 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:32.932350    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62" podStartSLOduration=1.932333917 podStartE2EDuration="1.932333917s" podCreationTimestamp="2026-05-25 13:16:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:16:32.931308942 +0000 UTC m=+708.934050694" watchObservedRunningTime="2026-05-25 13:16:32.932333917 +0000 UTC m=+708.935075670"
May 25 13:16:33.917678 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:33.917579    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"
May 25 13:16:33.918994 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:33.918964    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62" podUID="d968d149-a207-4af3-91bf-db1ae7d8b745" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 25 13:16:34.920995 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:34.920956    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62" podUID="d968d149-a207-4af3-91bf-db1ae7d8b745" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 25 13:16:37.882635 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:37.882580    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz" podUID="d8fe0506-8e9e-4e56-9b22-950b88f4e6f8" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:16:39.822548 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:39.822508    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln" podUID="ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 25 13:16:39.926647 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:39.926616    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"
May 25 13:16:39.927219 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:39.927192    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62" podUID="d968d149-a207-4af3-91bf-db1ae7d8b745" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 25 13:16:42.881205 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:42.881154    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz" podUID="d8fe0506-8e9e-4e56-9b22-950b88f4e6f8" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:16:42.881608 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:42.881288    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz"
May 25 13:16:47.881716 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:47.881673    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz" podUID="d8fe0506-8e9e-4e56-9b22-950b88f4e6f8" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:16:49.823628 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:49.823600    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"
May 25 13:16:49.927182 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:49.927136    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62" podUID="d968d149-a207-4af3-91bf-db1ae7d8b745" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 25 13:16:52.881841 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:52.881795    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz" podUID="d8fe0506-8e9e-4e56-9b22-950b88f4e6f8" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:16:57.881717 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:57.881675    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz" podUID="d8fe0506-8e9e-4e56-9b22-950b88f4e6f8" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:16:59.927316 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:16:59.927271    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62" podUID="d968d149-a207-4af3-91bf-db1ae7d8b745" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 25 13:17:00.851266 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:17:00.851228    2569 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8fe0506_8e9e_4e56_9b22_950b88f4e6f8.slice/crio-conmon-73c25c6532dc768daf5a7f7766503e02eef9922553698a801b5d0a25e50917ac.scope\": RecentStats: unable to find data in memory cache]"
May 25 13:17:00.851439 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:17:00.851339    2569 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8fe0506_8e9e_4e56_9b22_950b88f4e6f8.slice/crio-conmon-73c25c6532dc768daf5a7f7766503e02eef9922553698a801b5d0a25e50917ac.scope\": RecentStats: unable to find data in memory cache]"
May 25 13:17:00.960518 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:00.960493    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz"
May 25 13:17:01.008978 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.008947    2569 generic.go:358] "Generic (PLEG): container finished" podID="d8fe0506-8e9e-4e56-9b22-950b88f4e6f8" containerID="73c25c6532dc768daf5a7f7766503e02eef9922553698a801b5d0a25e50917ac" exitCode=0
May 25 13:17:01.009124 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.009014    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz"
May 25 13:17:01.009124 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.009031    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz" event={"ID":"d8fe0506-8e9e-4e56-9b22-950b88f4e6f8","Type":"ContainerDied","Data":"73c25c6532dc768daf5a7f7766503e02eef9922553698a801b5d0a25e50917ac"}
May 25 13:17:01.009124 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.009071    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz" event={"ID":"d8fe0506-8e9e-4e56-9b22-950b88f4e6f8","Type":"ContainerDied","Data":"e555ce8ea6b30e34b4da719480357629252f66ae3020c17d90e2dc025a24269c"}
May 25 13:17:01.009124 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.009086    2569 scope.go:117] "RemoveContainer" containerID="73c25c6532dc768daf5a7f7766503e02eef9922553698a801b5d0a25e50917ac"
May 25 13:17:01.017040 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.017016    2569 scope.go:117] "RemoveContainer" containerID="73c25c6532dc768daf5a7f7766503e02eef9922553698a801b5d0a25e50917ac"
May 25 13:17:01.017345 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:17:01.017308    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73c25c6532dc768daf5a7f7766503e02eef9922553698a801b5d0a25e50917ac\": container with ID starting with 73c25c6532dc768daf5a7f7766503e02eef9922553698a801b5d0a25e50917ac not found: ID does not exist" containerID="73c25c6532dc768daf5a7f7766503e02eef9922553698a801b5d0a25e50917ac"
May 25 13:17:01.017411 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.017357    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c25c6532dc768daf5a7f7766503e02eef9922553698a801b5d0a25e50917ac"} err="failed to get container status \"73c25c6532dc768daf5a7f7766503e02eef9922553698a801b5d0a25e50917ac\": rpc error: code = NotFound desc = could not find container \"73c25c6532dc768daf5a7f7766503e02eef9922553698a801b5d0a25e50917ac\": container with ID starting with 73c25c6532dc768daf5a7f7766503e02eef9922553698a801b5d0a25e50917ac not found: ID does not exist"
May 25 13:17:01.060401 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.060369    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8fe0506-8e9e-4e56-9b22-950b88f4e6f8-openshift-service-ca-bundle\") pod \"d8fe0506-8e9e-4e56-9b22-950b88f4e6f8\" (UID: \"d8fe0506-8e9e-4e56-9b22-950b88f4e6f8\") "
May 25 13:17:01.060561 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.060417    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8fe0506-8e9e-4e56-9b22-950b88f4e6f8-proxy-tls\") pod \"d8fe0506-8e9e-4e56-9b22-950b88f4e6f8\" (UID: \"d8fe0506-8e9e-4e56-9b22-950b88f4e6f8\") "
May 25 13:17:01.060797 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.060766    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8fe0506-8e9e-4e56-9b22-950b88f4e6f8-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d8fe0506-8e9e-4e56-9b22-950b88f4e6f8" (UID: "d8fe0506-8e9e-4e56-9b22-950b88f4e6f8"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:17:01.062448 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.062424    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8fe0506-8e9e-4e56-9b22-950b88f4e6f8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d8fe0506-8e9e-4e56-9b22-950b88f4e6f8" (UID: "d8fe0506-8e9e-4e56-9b22-950b88f4e6f8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:17:01.080575 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.080512    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s"]
May 25 13:17:01.080866 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.080854    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c58bfbf8-1df7-4757-9a67-89b8fb7745df" containerName="switch-graph-889c1"
May 25 13:17:01.080931 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.080868    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c58bfbf8-1df7-4757-9a67-89b8fb7745df" containerName="switch-graph-889c1"
May 25 13:17:01.080931 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.080886    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8fe0506-8e9e-4e56-9b22-950b88f4e6f8" containerName="model-chainer"
May 25 13:17:01.080931 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.080892    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8fe0506-8e9e-4e56-9b22-950b88f4e6f8" containerName="model-chainer"
May 25 13:17:01.081020 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.080947    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="c58bfbf8-1df7-4757-9a67-89b8fb7745df" containerName="switch-graph-889c1"
May 25 13:17:01.081020 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.080962    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="d8fe0506-8e9e-4e56-9b22-950b88f4e6f8" containerName="model-chainer"
May 25 13:17:01.085440 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.085423    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s"
May 25 13:17:01.087812 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.087791    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-339a4-serving-cert\""
May 25 13:17:01.087929 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.087815    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-339a4-kube-rbac-proxy-sar-config\""
May 25 13:17:01.091548 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.091515    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s"]
May 25 13:17:01.161864 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.161830    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8fe0506-8e9e-4e56-9b22-950b88f4e6f8-openshift-service-ca-bundle\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:17:01.161864 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.161860    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8fe0506-8e9e-4e56-9b22-950b88f4e6f8-proxy-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:17:01.262544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.262495    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/192093cf-923b-4607-8d51-3f0edab2a14e-proxy-tls\") pod \"switch-graph-339a4-68f7b7cdb-6xr4s\" (UID: \"192093cf-923b-4607-8d51-3f0edab2a14e\") " pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s"
May 25 13:17:01.262728 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.262582    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/192093cf-923b-4607-8d51-3f0edab2a14e-openshift-service-ca-bundle\") pod \"switch-graph-339a4-68f7b7cdb-6xr4s\" (UID: \"192093cf-923b-4607-8d51-3f0edab2a14e\") " pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s"
May 25 13:17:01.330156 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.330121    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz"]
May 25 13:17:01.335517 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.335439    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-866798f95b-9dhqz"]
May 25 13:17:01.363714 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.363672    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/192093cf-923b-4607-8d51-3f0edab2a14e-proxy-tls\") pod \"switch-graph-339a4-68f7b7cdb-6xr4s\" (UID: \"192093cf-923b-4607-8d51-3f0edab2a14e\") " pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s"
May 25 13:17:01.363910 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.363727    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/192093cf-923b-4607-8d51-3f0edab2a14e-openshift-service-ca-bundle\") pod \"switch-graph-339a4-68f7b7cdb-6xr4s\" (UID: \"192093cf-923b-4607-8d51-3f0edab2a14e\") " pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s"
May 25 13:17:01.364383 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.364361    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/192093cf-923b-4607-8d51-3f0edab2a14e-openshift-service-ca-bundle\") pod \"switch-graph-339a4-68f7b7cdb-6xr4s\" (UID: \"192093cf-923b-4607-8d51-3f0edab2a14e\") " pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s"
May 25 13:17:01.366230 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.366209    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/192093cf-923b-4607-8d51-3f0edab2a14e-proxy-tls\") pod \"switch-graph-339a4-68f7b7cdb-6xr4s\" (UID: \"192093cf-923b-4607-8d51-3f0edab2a14e\") " pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s"
May 25 13:17:01.397379 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.397341    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s"
May 25 13:17:01.524938 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:01.524906    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s"]
May 25 13:17:01.527448 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:17:01.527414    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192093cf_923b_4607_8d51_3f0edab2a14e.slice/crio-08d57240b83a9eac7f2006bd9df312041a66d58b37b6da06be4e446e24bb8620 WatchSource:0}: Error finding container 08d57240b83a9eac7f2006bd9df312041a66d58b37b6da06be4e446e24bb8620: Status 404 returned error can't find the container with id 08d57240b83a9eac7f2006bd9df312041a66d58b37b6da06be4e446e24bb8620
May 25 13:17:02.014792 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:02.014758    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s" event={"ID":"192093cf-923b-4607-8d51-3f0edab2a14e","Type":"ContainerStarted","Data":"bc86f568f66950cef8ca16d051544adebe30a8f377d38b53403c78075ff10f78"}
May 25 13:17:02.014792 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:02.014795    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s" event={"ID":"192093cf-923b-4607-8d51-3f0edab2a14e","Type":"ContainerStarted","Data":"08d57240b83a9eac7f2006bd9df312041a66d58b37b6da06be4e446e24bb8620"}
May 25 13:17:02.015246 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:02.014832    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s"
May 25 13:17:02.031420 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:02.031371    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s" podStartSLOduration=1.03135752 podStartE2EDuration="1.03135752s" podCreationTimestamp="2026-05-25 13:17:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:17:02.031276177 +0000 UTC m=+738.034017930" watchObservedRunningTime="2026-05-25 13:17:02.03135752 +0000 UTC m=+738.034099273"
May 25 13:17:02.602511 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:02.602478    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8fe0506-8e9e-4e56-9b22-950b88f4e6f8" path="/var/lib/kubelet/pods/d8fe0506-8e9e-4e56-9b22-950b88f4e6f8/volumes"
May 25 13:17:08.023718 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:08.023682    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s"
May 25 13:17:09.927327 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:09.927286    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62" podUID="d968d149-a207-4af3-91bf-db1ae7d8b745" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 25 13:17:19.927992 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:19.927957    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"
May 25 13:17:31.010041 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:31.010005    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d"]
May 25 13:17:31.013614 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:31.013590    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d"
May 25 13:17:31.016464 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:31.016443    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-08c99-serving-cert\""
May 25 13:17:31.016570 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:31.016445    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-08c99-kube-rbac-proxy-sar-config\""
May 25 13:17:31.022417 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:31.022390    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d"]
May 25 13:17:31.111995 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:31.111966    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c713f40d-97e5-4aef-9cde-11fcce6f1ec4-openshift-service-ca-bundle\") pod \"sequence-graph-08c99-84bc9f87db-h8b2d\" (UID: \"c713f40d-97e5-4aef-9cde-11fcce6f1ec4\") " pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d"
May 25 13:17:31.111995 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:31.112002    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c713f40d-97e5-4aef-9cde-11fcce6f1ec4-proxy-tls\") pod \"sequence-graph-08c99-84bc9f87db-h8b2d\" (UID: \"c713f40d-97e5-4aef-9cde-11fcce6f1ec4\") " pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d"
May 25 13:17:31.213266 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:31.213224    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c713f40d-97e5-4aef-9cde-11fcce6f1ec4-openshift-service-ca-bundle\") pod \"sequence-graph-08c99-84bc9f87db-h8b2d\" (UID: \"c713f40d-97e5-4aef-9cde-11fcce6f1ec4\") " pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d"
May 25 13:17:31.213442 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:31.213277    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c713f40d-97e5-4aef-9cde-11fcce6f1ec4-proxy-tls\") pod \"sequence-graph-08c99-84bc9f87db-h8b2d\" (UID: \"c713f40d-97e5-4aef-9cde-11fcce6f1ec4\") " pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d"
May 25 13:17:31.213899 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:31.213877    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c713f40d-97e5-4aef-9cde-11fcce6f1ec4-openshift-service-ca-bundle\") pod \"sequence-graph-08c99-84bc9f87db-h8b2d\" (UID: \"c713f40d-97e5-4aef-9cde-11fcce6f1ec4\") " pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d"
May 25 13:17:31.215642 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:31.215625    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c713f40d-97e5-4aef-9cde-11fcce6f1ec4-proxy-tls\") pod \"sequence-graph-08c99-84bc9f87db-h8b2d\" (UID: \"c713f40d-97e5-4aef-9cde-11fcce6f1ec4\") " pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d"
May 25 13:17:31.326741 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:31.326652    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d"
May 25 13:17:31.461419 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:31.461388    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d"]
May 25 13:17:31.463720 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:17:31.463690    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc713f40d_97e5_4aef_9cde_11fcce6f1ec4.slice/crio-4cb87a8c759478161ce0f8a54ac4e3a51565ae80d27d9c8f8e93298bccecf2b2 WatchSource:0}: Error finding container 4cb87a8c759478161ce0f8a54ac4e3a51565ae80d27d9c8f8e93298bccecf2b2: Status 404 returned error can't find the container with id 4cb87a8c759478161ce0f8a54ac4e3a51565ae80d27d9c8f8e93298bccecf2b2
May 25 13:17:32.119448 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:32.119413    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d" event={"ID":"c713f40d-97e5-4aef-9cde-11fcce6f1ec4","Type":"ContainerStarted","Data":"fe1f8cbf0c3788995178478b0c4613691fcffb4716b452841e88a6a7c434fd24"}
May 25 13:17:32.119448 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:32.119449    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d" event={"ID":"c713f40d-97e5-4aef-9cde-11fcce6f1ec4","Type":"ContainerStarted","Data":"4cb87a8c759478161ce0f8a54ac4e3a51565ae80d27d9c8f8e93298bccecf2b2"}
May 25 13:17:32.119881 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:32.119591    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d"
May 25 13:17:32.137724 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:32.137675    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d" podStartSLOduration=2.137659192 podStartE2EDuration="2.137659192s" podCreationTimestamp="2026-05-25 13:17:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:17:32.135645954 +0000 UTC m=+768.138387717" watchObservedRunningTime="2026-05-25 13:17:32.137659192 +0000 UTC m=+768.140400945"
May 25 13:17:38.128744 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:17:38.128708    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d"
May 25 13:19:45.572194 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:19:45.572144    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/1.log"
May 25 13:19:45.572711 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:19:45.572264    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/1.log"
May 25 13:19:45.576704 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:19:45.576684    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:19:45.576892 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:19:45.576879    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:24:45.597875 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:24:45.597789    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/1.log"
May 25 13:24:45.600144 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:24:45.600116    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/1.log"
May 25 13:24:45.602937 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:24:45.602920    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:24:45.604789 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:24:45.604770    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:25:15.663341 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:15.663303    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s"]
May 25 13:25:15.663909 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:15.663560    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s" podUID="192093cf-923b-4607-8d51-3f0edab2a14e" containerName="switch-graph-339a4" containerID="cri-o://bc86f568f66950cef8ca16d051544adebe30a8f377d38b53403c78075ff10f78" gracePeriod=30
May 25 13:25:15.794658 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:15.794624    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"]
May 25 13:25:15.794944 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:15.794919    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln" podUID="ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb" containerName="kserve-container" containerID="cri-o://8544c3b2807283bedf8fca3e2d9c5eee08673476c898a9a01db2a56075f900b3" gracePeriod=30
May 25 13:25:15.795028 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:15.794971    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln" podUID="ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb" containerName="kube-rbac-proxy" containerID="cri-o://f4cef4a02c7ccc0abab1b91b7bb5724cd17527c0bcba98823a520716a243e18d" gracePeriod=30
May 25 13:25:15.919401 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:15.919322    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"]
May 25 13:25:15.922922 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:15.922897    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"
May 25 13:25:15.925246 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:15.925225    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-db114-predictor-serving-cert\""
May 25 13:25:15.925374 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:15.925243    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-db114-kube-rbac-proxy-sar-config\""
May 25 13:25:15.932777 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:15.932746    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"]
May 25 13:25:15.991105 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:15.991069    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-db114-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de4f786e-ee21-4acf-a660-ec364cf8ac74-error-404-isvc-db114-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-db114-predictor-7d7bc84957-tkjp8\" (UID: \"de4f786e-ee21-4acf-a660-ec364cf8ac74\") " pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"
May 25 13:25:15.991310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:15.991120    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de4f786e-ee21-4acf-a660-ec364cf8ac74-proxy-tls\") pod \"error-404-isvc-db114-predictor-7d7bc84957-tkjp8\" (UID: \"de4f786e-ee21-4acf-a660-ec364cf8ac74\") " pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"
May 25 13:25:15.991310 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:15.991193    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp4gt\" (UniqueName: \"kubernetes.io/projected/de4f786e-ee21-4acf-a660-ec364cf8ac74-kube-api-access-hp4gt\") pod \"error-404-isvc-db114-predictor-7d7bc84957-tkjp8\" (UID: \"de4f786e-ee21-4acf-a660-ec364cf8ac74\") " pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"
May 25 13:25:16.091996 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:16.091955    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-db114-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de4f786e-ee21-4acf-a660-ec364cf8ac74-error-404-isvc-db114-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-db114-predictor-7d7bc84957-tkjp8\" (UID: \"de4f786e-ee21-4acf-a660-ec364cf8ac74\") " pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"
May 25 13:25:16.092229 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:16.092019    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de4f786e-ee21-4acf-a660-ec364cf8ac74-proxy-tls\") pod \"error-404-isvc-db114-predictor-7d7bc84957-tkjp8\" (UID: \"de4f786e-ee21-4acf-a660-ec364cf8ac74\") " pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"
May 25 13:25:16.092229 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:16.092064    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hp4gt\" (UniqueName: \"kubernetes.io/projected/de4f786e-ee21-4acf-a660-ec364cf8ac74-kube-api-access-hp4gt\") pod \"error-404-isvc-db114-predictor-7d7bc84957-tkjp8\" (UID: \"de4f786e-ee21-4acf-a660-ec364cf8ac74\") " pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"
May 25 13:25:16.092229 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:25:16.092116    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-db114-predictor-serving-cert: secret "error-404-isvc-db114-predictor-serving-cert" not found
May 25 13:25:16.092229 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:25:16.092216    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de4f786e-ee21-4acf-a660-ec364cf8ac74-proxy-tls podName:de4f786e-ee21-4acf-a660-ec364cf8ac74 nodeName:}" failed. No retries permitted until 2026-05-25 13:25:16.592163497 +0000 UTC m=+1232.594905248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/de4f786e-ee21-4acf-a660-ec364cf8ac74-proxy-tls") pod "error-404-isvc-db114-predictor-7d7bc84957-tkjp8" (UID: "de4f786e-ee21-4acf-a660-ec364cf8ac74") : secret "error-404-isvc-db114-predictor-serving-cert" not found
May 25 13:25:16.092781 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:16.092757    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-db114-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de4f786e-ee21-4acf-a660-ec364cf8ac74-error-404-isvc-db114-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-db114-predictor-7d7bc84957-tkjp8\" (UID: \"de4f786e-ee21-4acf-a660-ec364cf8ac74\") " pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"
May 25 13:25:16.100708 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:16.100687    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp4gt\" (UniqueName: \"kubernetes.io/projected/de4f786e-ee21-4acf-a660-ec364cf8ac74-kube-api-access-hp4gt\") pod \"error-404-isvc-db114-predictor-7d7bc84957-tkjp8\" (UID: \"de4f786e-ee21-4acf-a660-ec364cf8ac74\") " pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"
May 25 13:25:16.596271 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:16.596236    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de4f786e-ee21-4acf-a660-ec364cf8ac74-proxy-tls\") pod \"error-404-isvc-db114-predictor-7d7bc84957-tkjp8\" (UID: \"de4f786e-ee21-4acf-a660-ec364cf8ac74\") " pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"
May 25 13:25:16.598685 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:16.598657    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de4f786e-ee21-4acf-a660-ec364cf8ac74-proxy-tls\") pod \"error-404-isvc-db114-predictor-7d7bc84957-tkjp8\" (UID: \"de4f786e-ee21-4acf-a660-ec364cf8ac74\") " pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"
May 25 13:25:16.697330 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:16.697298    2569 generic.go:358] "Generic (PLEG): container finished" podID="ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb" containerID="f4cef4a02c7ccc0abab1b91b7bb5724cd17527c0bcba98823a520716a243e18d" exitCode=2
May 25 13:25:16.697721 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:16.697360    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln" event={"ID":"ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb","Type":"ContainerDied","Data":"f4cef4a02c7ccc0abab1b91b7bb5724cd17527c0bcba98823a520716a243e18d"}
May 25 13:25:16.836993 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:16.836956    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"
May 25 13:25:16.967715 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:16.967688    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"]
May 25 13:25:16.970258 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:25:16.970229    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde4f786e_ee21_4acf_a660_ec364cf8ac74.slice/crio-c67e0d8b528c3c97fbb00bfd356286ca55bb1994289c037dccfa8302c1795956 WatchSource:0}: Error finding container c67e0d8b528c3c97fbb00bfd356286ca55bb1994289c037dccfa8302c1795956: Status 404 returned error can't find the container with id c67e0d8b528c3c97fbb00bfd356286ca55bb1994289c037dccfa8302c1795956
May 25 13:25:16.972083 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:16.972068    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:25:17.702129 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:17.702088    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8" event={"ID":"de4f786e-ee21-4acf-a660-ec364cf8ac74","Type":"ContainerStarted","Data":"3c7856ab965df4f41796ba86e78e13fd50bc154fabc04cd72bba90b045532989"}
May 25 13:25:17.702526 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:17.702136    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8" event={"ID":"de4f786e-ee21-4acf-a660-ec364cf8ac74","Type":"ContainerStarted","Data":"4ba6a44912e6f4d0dd1f373c486eb2b2f2c265e7e5b78a40959b2498b907a139"}
May 25 13:25:17.702526 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:17.702152    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8" event={"ID":"de4f786e-ee21-4acf-a660-ec364cf8ac74","Type":"ContainerStarted","Data":"c67e0d8b528c3c97fbb00bfd356286ca55bb1994289c037dccfa8302c1795956"}
May 25 13:25:17.702526 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:17.702292    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"
May 25 13:25:17.702526 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:17.702420    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"
May 25 13:25:17.703605 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:17.703585    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8" podUID="de4f786e-ee21-4acf-a660-ec364cf8ac74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 13:25:17.724872 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:17.724820    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8" podStartSLOduration=2.724807702 podStartE2EDuration="2.724807702s" podCreationTimestamp="2026-05-25 13:25:15 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:25:17.723440051 +0000 UTC m=+1233.726181803" watchObservedRunningTime="2026-05-25 13:25:17.724807702 +0000 UTC m=+1233.727549453"
May 25 13:25:18.022128 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:18.022033    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s" podUID="192093cf-923b-4607-8d51-3f0edab2a14e" containerName="switch-graph-339a4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:25:18.705257 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:18.705215    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8" podUID="de4f786e-ee21-4acf-a660-ec364cf8ac74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 13:25:19.345813 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:19.345789    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"
May 25 13:25:19.422782 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:19.422684    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-339a4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb-error-404-isvc-339a4-kube-rbac-proxy-sar-config\") pod \"ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb\" (UID: \"ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb\") "
May 25 13:25:19.422782 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:19.422765    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmdmx\" (UniqueName: \"kubernetes.io/projected/ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb-kube-api-access-xmdmx\") pod \"ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb\" (UID: \"ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb\") "
May 25 13:25:19.422966 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:19.422800    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb-proxy-tls\") pod \"ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb\" (UID: \"ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb\") "
May 25 13:25:19.423120 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:19.423090    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb-error-404-isvc-339a4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-339a4-kube-rbac-proxy-sar-config") pod "ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb" (UID: "ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb"). InnerVolumeSpecName "error-404-isvc-339a4-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:25:19.424983 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:19.424954    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb" (UID: "ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:25:19.425101 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:19.424963    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb-kube-api-access-xmdmx" (OuterVolumeSpecName: "kube-api-access-xmdmx") pod "ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb" (UID: "ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb"). InnerVolumeSpecName "kube-api-access-xmdmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:25:19.523891 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:19.523851    2569 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-339a4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb-error-404-isvc-339a4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:25:19.523891 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:19.523884    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmdmx\" (UniqueName: \"kubernetes.io/projected/ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb-kube-api-access-xmdmx\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:25:19.523891 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:19.523894    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb-proxy-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:25:19.709955 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:19.709856    2569 generic.go:358] "Generic (PLEG): container finished" podID="ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb" containerID="8544c3b2807283bedf8fca3e2d9c5eee08673476c898a9a01db2a56075f900b3" exitCode=0
May 25 13:25:19.709955 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:19.709926    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"
May 25 13:25:19.709955 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:19.709946    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln" event={"ID":"ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb","Type":"ContainerDied","Data":"8544c3b2807283bedf8fca3e2d9c5eee08673476c898a9a01db2a56075f900b3"}
May 25 13:25:19.710464 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:19.709985    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln" event={"ID":"ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb","Type":"ContainerDied","Data":"e31d233fb0313d932a7b06e7ef378371cea355c8322f441407a343db99bd2a58"}
May 25 13:25:19.710464 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:19.710000    2569 scope.go:117] "RemoveContainer" containerID="f4cef4a02c7ccc0abab1b91b7bb5724cd17527c0bcba98823a520716a243e18d"
May 25 13:25:19.720579 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:19.720554    2569 scope.go:117] "RemoveContainer" containerID="8544c3b2807283bedf8fca3e2d9c5eee08673476c898a9a01db2a56075f900b3"
May 25 13:25:19.729057 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:19.728905    2569 scope.go:117] "RemoveContainer" containerID="f4cef4a02c7ccc0abab1b91b7bb5724cd17527c0bcba98823a520716a243e18d"
May 25 13:25:19.729308 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:25:19.729254    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4cef4a02c7ccc0abab1b91b7bb5724cd17527c0bcba98823a520716a243e18d\": container with ID starting with f4cef4a02c7ccc0abab1b91b7bb5724cd17527c0bcba98823a520716a243e18d not found: ID does not exist" containerID="f4cef4a02c7ccc0abab1b91b7bb5724cd17527c0bcba98823a520716a243e18d"
May 25 13:25:19.729308 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:19.729292    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4cef4a02c7ccc0abab1b91b7bb5724cd17527c0bcba98823a520716a243e18d"} err="failed to get container status \"f4cef4a02c7ccc0abab1b91b7bb5724cd17527c0bcba98823a520716a243e18d\": rpc error: code = NotFound desc = could not find container \"f4cef4a02c7ccc0abab1b91b7bb5724cd17527c0bcba98823a520716a243e18d\": container with ID starting with f4cef4a02c7ccc0abab1b91b7bb5724cd17527c0bcba98823a520716a243e18d not found: ID does not exist"
May 25 13:25:19.729473 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:19.729319    2569 scope.go:117] "RemoveContainer" containerID="8544c3b2807283bedf8fca3e2d9c5eee08673476c898a9a01db2a56075f900b3"
May 25 13:25:19.730416 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:25:19.730371    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8544c3b2807283bedf8fca3e2d9c5eee08673476c898a9a01db2a56075f900b3\": container with ID starting with 8544c3b2807283bedf8fca3e2d9c5eee08673476c898a9a01db2a56075f900b3 not found: ID does not exist" containerID="8544c3b2807283bedf8fca3e2d9c5eee08673476c898a9a01db2a56075f900b3"
May 25 13:25:19.730416 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:19.730407    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8544c3b2807283bedf8fca3e2d9c5eee08673476c898a9a01db2a56075f900b3"} err="failed to get container status \"8544c3b2807283bedf8fca3e2d9c5eee08673476c898a9a01db2a56075f900b3\": rpc error: code = NotFound desc = could not find container \"8544c3b2807283bedf8fca3e2d9c5eee08673476c898a9a01db2a56075f900b3\": container with ID starting with 8544c3b2807283bedf8fca3e2d9c5eee08673476c898a9a01db2a56075f900b3 not found: ID does not exist"
May 25 13:25:19.731655 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:19.731631    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"]
May 25 13:25:19.735444 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:19.735420    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-339a4-predictor-7cdc89ff49-ncgln"]
May 25 13:25:20.597468 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:20.597435    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb" path="/var/lib/kubelet/pods/ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb/volumes"
May 25 13:25:23.022623 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:23.022575    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s" podUID="192093cf-923b-4607-8d51-3f0edab2a14e" containerName="switch-graph-339a4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:25:23.710244 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:23.710214    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"
May 25 13:25:23.710643 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:23.710607    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8" podUID="de4f786e-ee21-4acf-a660-ec364cf8ac74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 13:25:28.022439 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:28.022397    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s" podUID="192093cf-923b-4607-8d51-3f0edab2a14e" containerName="switch-graph-339a4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:25:28.022912 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:28.022529    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s"
May 25 13:25:33.021925 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:33.021879    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s" podUID="192093cf-923b-4607-8d51-3f0edab2a14e" containerName="switch-graph-339a4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:25:33.710849 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:33.710767    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8" podUID="de4f786e-ee21-4acf-a660-ec364cf8ac74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 13:25:38.021923 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:38.021880    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s" podUID="192093cf-923b-4607-8d51-3f0edab2a14e" containerName="switch-graph-339a4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:25:43.022807 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:43.022762    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s" podUID="192093cf-923b-4607-8d51-3f0edab2a14e" containerName="switch-graph-339a4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:25:43.710773 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:43.710731    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8" podUID="de4f786e-ee21-4acf-a660-ec364cf8ac74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 13:25:45.746573 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:45.746517    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d"]
May 25 13:25:45.746926 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:45.746843    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d" podUID="c713f40d-97e5-4aef-9cde-11fcce6f1ec4" containerName="sequence-graph-08c99" containerID="cri-o://fe1f8cbf0c3788995178478b0c4613691fcffb4716b452841e88a6a7c434fd24" gracePeriod=30
May 25 13:25:45.800829 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:45.800797    2569 generic.go:358] "Generic (PLEG): container finished" podID="192093cf-923b-4607-8d51-3f0edab2a14e" containerID="bc86f568f66950cef8ca16d051544adebe30a8f377d38b53403c78075ff10f78" exitCode=0
May 25 13:25:45.801004 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:45.800870    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s" event={"ID":"192093cf-923b-4607-8d51-3f0edab2a14e","Type":"ContainerDied","Data":"bc86f568f66950cef8ca16d051544adebe30a8f377d38b53403c78075ff10f78"}
May 25 13:25:45.828751 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:45.828725    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s"
May 25 13:25:45.848269 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:45.848206    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/192093cf-923b-4607-8d51-3f0edab2a14e-openshift-service-ca-bundle\") pod \"192093cf-923b-4607-8d51-3f0edab2a14e\" (UID: \"192093cf-923b-4607-8d51-3f0edab2a14e\") "
May 25 13:25:45.848461 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:45.848331    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/192093cf-923b-4607-8d51-3f0edab2a14e-proxy-tls\") pod \"192093cf-923b-4607-8d51-3f0edab2a14e\" (UID: \"192093cf-923b-4607-8d51-3f0edab2a14e\") "
May 25 13:25:45.848804 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:45.848620    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192093cf-923b-4607-8d51-3f0edab2a14e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "192093cf-923b-4607-8d51-3f0edab2a14e" (UID: "192093cf-923b-4607-8d51-3f0edab2a14e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:25:45.848804 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:45.848777    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/192093cf-923b-4607-8d51-3f0edab2a14e-openshift-service-ca-bundle\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:25:45.851626 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:45.851417    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/192093cf-923b-4607-8d51-3f0edab2a14e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "192093cf-923b-4607-8d51-3f0edab2a14e" (UID: "192093cf-923b-4607-8d51-3f0edab2a14e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:25:45.912831 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:45.912795    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"]
May 25 13:25:45.913112 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:45.913080    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62" podUID="d968d149-a207-4af3-91bf-db1ae7d8b745" containerName="kserve-container" containerID="cri-o://231818c82ca9a23c34f94fcc9e8f2068d8322f3d8dd0620000edd9bb7c9fe243" gracePeriod=30
May 25 13:25:45.913282 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:45.913138    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62" podUID="d968d149-a207-4af3-91bf-db1ae7d8b745" containerName="kube-rbac-proxy" containerID="cri-o://6b565544c019ce6e9f75b2aca9c3e168e47595e8965f72eddda905d2b8e8014d" gracePeriod=30
May 25 13:25:45.950210 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:45.950157    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/192093cf-923b-4607-8d51-3f0edab2a14e-proxy-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:25:46.074348 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.074312    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"]
May 25 13:25:46.074874 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.074860    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb" containerName="kube-rbac-proxy"
May 25 13:25:46.074917 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.074878    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb" containerName="kube-rbac-proxy"
May 25 13:25:46.074917 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.074913    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="192093cf-923b-4607-8d51-3f0edab2a14e" containerName="switch-graph-339a4"
May 25 13:25:46.074982 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.074922    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="192093cf-923b-4607-8d51-3f0edab2a14e" containerName="switch-graph-339a4"
May 25 13:25:46.074982 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.074935    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb" containerName="kserve-container"
May 25 13:25:46.074982 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.074944    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb" containerName="kserve-container"
May 25 13:25:46.075084 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.075021    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb" containerName="kube-rbac-proxy"
May 25 13:25:46.075084 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.075033    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="192093cf-923b-4607-8d51-3f0edab2a14e" containerName="switch-graph-339a4"
May 25 13:25:46.075084 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.075044    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="ff3eadb9-a8c4-43e4-bee2-4e3e1727f2cb" containerName="kserve-container"
May 25 13:25:46.078599 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.078578    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"
May 25 13:25:46.081538 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.081514    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9ee57-kube-rbac-proxy-sar-config\""
May 25 13:25:46.082029 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.082012    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9ee57-predictor-serving-cert\""
May 25 13:25:46.091205 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.091159    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"]
May 25 13:25:46.151559 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.151524    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-9ee57-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce196080-f42b-4918-b6f6-f42da3537e67-error-404-isvc-9ee57-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9ee57-predictor-d974ddf95-6jxwb\" (UID: \"ce196080-f42b-4918-b6f6-f42da3537e67\") " pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"
May 25 13:25:46.151734 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.151572    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce196080-f42b-4918-b6f6-f42da3537e67-proxy-tls\") pod \"error-404-isvc-9ee57-predictor-d974ddf95-6jxwb\" (UID: \"ce196080-f42b-4918-b6f6-f42da3537e67\") " pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"
May 25 13:25:46.151734 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.151680    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzp98\" (UniqueName: \"kubernetes.io/projected/ce196080-f42b-4918-b6f6-f42da3537e67-kube-api-access-vzp98\") pod \"error-404-isvc-9ee57-predictor-d974ddf95-6jxwb\" (UID: \"ce196080-f42b-4918-b6f6-f42da3537e67\") " pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"
May 25 13:25:46.252651 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.252563    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-9ee57-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce196080-f42b-4918-b6f6-f42da3537e67-error-404-isvc-9ee57-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9ee57-predictor-d974ddf95-6jxwb\" (UID: \"ce196080-f42b-4918-b6f6-f42da3537e67\") " pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"
May 25 13:25:46.252651 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.252620    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce196080-f42b-4918-b6f6-f42da3537e67-proxy-tls\") pod \"error-404-isvc-9ee57-predictor-d974ddf95-6jxwb\" (UID: \"ce196080-f42b-4918-b6f6-f42da3537e67\") " pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"
May 25 13:25:46.252861 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.252685    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vzp98\" (UniqueName: \"kubernetes.io/projected/ce196080-f42b-4918-b6f6-f42da3537e67-kube-api-access-vzp98\") pod \"error-404-isvc-9ee57-predictor-d974ddf95-6jxwb\" (UID: \"ce196080-f42b-4918-b6f6-f42da3537e67\") " pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"
May 25 13:25:46.252861 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:25:46.252798    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-serving-cert: secret "error-404-isvc-9ee57-predictor-serving-cert" not found
May 25 13:25:46.252946 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:25:46.252866    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce196080-f42b-4918-b6f6-f42da3537e67-proxy-tls podName:ce196080-f42b-4918-b6f6-f42da3537e67 nodeName:}" failed. No retries permitted until 2026-05-25 13:25:46.752848155 +0000 UTC m=+1262.755589887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ce196080-f42b-4918-b6f6-f42da3537e67-proxy-tls") pod "error-404-isvc-9ee57-predictor-d974ddf95-6jxwb" (UID: "ce196080-f42b-4918-b6f6-f42da3537e67") : secret "error-404-isvc-9ee57-predictor-serving-cert" not found
May 25 13:25:46.253335 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.253316    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-9ee57-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce196080-f42b-4918-b6f6-f42da3537e67-error-404-isvc-9ee57-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9ee57-predictor-d974ddf95-6jxwb\" (UID: \"ce196080-f42b-4918-b6f6-f42da3537e67\") " pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"
May 25 13:25:46.263408 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.263375    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzp98\" (UniqueName: \"kubernetes.io/projected/ce196080-f42b-4918-b6f6-f42da3537e67-kube-api-access-vzp98\") pod \"error-404-isvc-9ee57-predictor-d974ddf95-6jxwb\" (UID: \"ce196080-f42b-4918-b6f6-f42da3537e67\") " pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"
May 25 13:25:46.758442 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.758393    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce196080-f42b-4918-b6f6-f42da3537e67-proxy-tls\") pod \"error-404-isvc-9ee57-predictor-d974ddf95-6jxwb\" (UID: \"ce196080-f42b-4918-b6f6-f42da3537e67\") " pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"
May 25 13:25:46.760821 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.760795    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce196080-f42b-4918-b6f6-f42da3537e67-proxy-tls\") pod \"error-404-isvc-9ee57-predictor-d974ddf95-6jxwb\" (UID: \"ce196080-f42b-4918-b6f6-f42da3537e67\") " pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"
May 25 13:25:46.806651 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.806613    2569 generic.go:358] "Generic (PLEG): container finished" podID="d968d149-a207-4af3-91bf-db1ae7d8b745" containerID="6b565544c019ce6e9f75b2aca9c3e168e47595e8965f72eddda905d2b8e8014d" exitCode=2
May 25 13:25:46.806834 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.806684    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62" event={"ID":"d968d149-a207-4af3-91bf-db1ae7d8b745","Type":"ContainerDied","Data":"6b565544c019ce6e9f75b2aca9c3e168e47595e8965f72eddda905d2b8e8014d"}
May 25 13:25:46.807734 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.807709    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s" event={"ID":"192093cf-923b-4607-8d51-3f0edab2a14e","Type":"ContainerDied","Data":"08d57240b83a9eac7f2006bd9df312041a66d58b37b6da06be4e446e24bb8620"}
May 25 13:25:46.807734 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.807722    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s"
May 25 13:25:46.807903 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.807743    2569 scope.go:117] "RemoveContainer" containerID="bc86f568f66950cef8ca16d051544adebe30a8f377d38b53403c78075ff10f78"
May 25 13:25:46.823802 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.823772    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s"]
May 25 13:25:46.827020 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.826993    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-339a4-68f7b7cdb-6xr4s"]
May 25 13:25:46.990930 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:46.990835    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"
May 25 13:25:47.130250 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:47.130215    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"]
May 25 13:25:47.133912 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:25:47.133876    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce196080_f42b_4918_b6f6_f42da3537e67.slice/crio-663e2f5315b1291867fd6ec4171cda10b879ddec45891b8b2852fb6094e6dd96 WatchSource:0}: Error finding container 663e2f5315b1291867fd6ec4171cda10b879ddec45891b8b2852fb6094e6dd96: Status 404 returned error can't find the container with id 663e2f5315b1291867fd6ec4171cda10b879ddec45891b8b2852fb6094e6dd96
May 25 13:25:47.817324 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:47.817286    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb" event={"ID":"ce196080-f42b-4918-b6f6-f42da3537e67","Type":"ContainerStarted","Data":"f22e6dd75d905aa5162b4e376ed698e2e910c2f108542f31572436547472c40a"}
May 25 13:25:47.817324 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:47.817329    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb" event={"ID":"ce196080-f42b-4918-b6f6-f42da3537e67","Type":"ContainerStarted","Data":"88a6c18d2e11e2450940016d5d8af3f158c5e7f1089602d9399f6bc83df99fd4"}
May 25 13:25:47.817965 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:47.817345    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb" event={"ID":"ce196080-f42b-4918-b6f6-f42da3537e67","Type":"ContainerStarted","Data":"663e2f5315b1291867fd6ec4171cda10b879ddec45891b8b2852fb6094e6dd96"}
May 25 13:25:47.817965 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:47.817377    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"
May 25 13:25:47.836402 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:47.836342    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb" podStartSLOduration=1.836321568 podStartE2EDuration="1.836321568s" podCreationTimestamp="2026-05-25 13:25:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:25:47.833896241 +0000 UTC m=+1263.836637994" watchObservedRunningTime="2026-05-25 13:25:47.836321568 +0000 UTC m=+1263.839063322"
May 25 13:25:48.127937 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:48.127889    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d" podUID="c713f40d-97e5-4aef-9cde-11fcce6f1ec4" containerName="sequence-graph-08c99" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:25:48.597157 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:48.597120    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="192093cf-923b-4607-8d51-3f0edab2a14e" path="/var/lib/kubelet/pods/192093cf-923b-4607-8d51-3f0edab2a14e/volumes"
May 25 13:25:48.820325 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:48.820292    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"
May 25 13:25:48.821746 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:48.821716    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb" podUID="ce196080-f42b-4918-b6f6-f42da3537e67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 25 13:25:49.672736 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.672709    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"
May 25 13:25:49.785436 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.785343    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d968d149-a207-4af3-91bf-db1ae7d8b745-proxy-tls\") pod \"d968d149-a207-4af3-91bf-db1ae7d8b745\" (UID: \"d968d149-a207-4af3-91bf-db1ae7d8b745\") "
May 25 13:25:49.785436 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.785416    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdnnx\" (UniqueName: \"kubernetes.io/projected/d968d149-a207-4af3-91bf-db1ae7d8b745-kube-api-access-pdnnx\") pod \"d968d149-a207-4af3-91bf-db1ae7d8b745\" (UID: \"d968d149-a207-4af3-91bf-db1ae7d8b745\") "
May 25 13:25:49.785648 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.785444    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-08c99-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d968d149-a207-4af3-91bf-db1ae7d8b745-error-404-isvc-08c99-kube-rbac-proxy-sar-config\") pod \"d968d149-a207-4af3-91bf-db1ae7d8b745\" (UID: \"d968d149-a207-4af3-91bf-db1ae7d8b745\") "
May 25 13:25:49.785930 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.785856    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d968d149-a207-4af3-91bf-db1ae7d8b745-error-404-isvc-08c99-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-08c99-kube-rbac-proxy-sar-config") pod "d968d149-a207-4af3-91bf-db1ae7d8b745" (UID: "d968d149-a207-4af3-91bf-db1ae7d8b745"). InnerVolumeSpecName "error-404-isvc-08c99-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:25:49.787537 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.787508    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d968d149-a207-4af3-91bf-db1ae7d8b745-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d968d149-a207-4af3-91bf-db1ae7d8b745" (UID: "d968d149-a207-4af3-91bf-db1ae7d8b745"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:25:49.787654 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.787619    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d968d149-a207-4af3-91bf-db1ae7d8b745-kube-api-access-pdnnx" (OuterVolumeSpecName: "kube-api-access-pdnnx") pod "d968d149-a207-4af3-91bf-db1ae7d8b745" (UID: "d968d149-a207-4af3-91bf-db1ae7d8b745"). InnerVolumeSpecName "kube-api-access-pdnnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:25:49.825155 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.825119    2569 generic.go:358] "Generic (PLEG): container finished" podID="d968d149-a207-4af3-91bf-db1ae7d8b745" containerID="231818c82ca9a23c34f94fcc9e8f2068d8322f3d8dd0620000edd9bb7c9fe243" exitCode=0
May 25 13:25:49.825606 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.825223    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62" event={"ID":"d968d149-a207-4af3-91bf-db1ae7d8b745","Type":"ContainerDied","Data":"231818c82ca9a23c34f94fcc9e8f2068d8322f3d8dd0620000edd9bb7c9fe243"}
May 25 13:25:49.825606 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.825276    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62" event={"ID":"d968d149-a207-4af3-91bf-db1ae7d8b745","Type":"ContainerDied","Data":"96451f3a8e9a7728e48e5023d2cf75b06b07fb548d471f640238daa39d927911"}
May 25 13:25:49.825606 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.825242    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"
May 25 13:25:49.825606 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.825301    2569 scope.go:117] "RemoveContainer" containerID="6b565544c019ce6e9f75b2aca9c3e168e47595e8965f72eddda905d2b8e8014d"
May 25 13:25:49.826066 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.826038    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb" podUID="ce196080-f42b-4918-b6f6-f42da3537e67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 25 13:25:49.834322 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.834297    2569 scope.go:117] "RemoveContainer" containerID="231818c82ca9a23c34f94fcc9e8f2068d8322f3d8dd0620000edd9bb7c9fe243"
May 25 13:25:49.842738 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.842714    2569 scope.go:117] "RemoveContainer" containerID="6b565544c019ce6e9f75b2aca9c3e168e47595e8965f72eddda905d2b8e8014d"
May 25 13:25:49.843041 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:25:49.843018    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b565544c019ce6e9f75b2aca9c3e168e47595e8965f72eddda905d2b8e8014d\": container with ID starting with 6b565544c019ce6e9f75b2aca9c3e168e47595e8965f72eddda905d2b8e8014d not found: ID does not exist" containerID="6b565544c019ce6e9f75b2aca9c3e168e47595e8965f72eddda905d2b8e8014d"
May 25 13:25:49.843115 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.843049    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b565544c019ce6e9f75b2aca9c3e168e47595e8965f72eddda905d2b8e8014d"} err="failed to get container status \"6b565544c019ce6e9f75b2aca9c3e168e47595e8965f72eddda905d2b8e8014d\": rpc error: code = NotFound desc = could not find container \"6b565544c019ce6e9f75b2aca9c3e168e47595e8965f72eddda905d2b8e8014d\": container with ID starting with 6b565544c019ce6e9f75b2aca9c3e168e47595e8965f72eddda905d2b8e8014d not found: ID does not exist"
May 25 13:25:49.843115 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.843068    2569 scope.go:117] "RemoveContainer" containerID="231818c82ca9a23c34f94fcc9e8f2068d8322f3d8dd0620000edd9bb7c9fe243"
May 25 13:25:49.843334 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:25:49.843318    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231818c82ca9a23c34f94fcc9e8f2068d8322f3d8dd0620000edd9bb7c9fe243\": container with ID starting with 231818c82ca9a23c34f94fcc9e8f2068d8322f3d8dd0620000edd9bb7c9fe243 not found: ID does not exist" containerID="231818c82ca9a23c34f94fcc9e8f2068d8322f3d8dd0620000edd9bb7c9fe243"
May 25 13:25:49.843381 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.843341    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231818c82ca9a23c34f94fcc9e8f2068d8322f3d8dd0620000edd9bb7c9fe243"} err="failed to get container status \"231818c82ca9a23c34f94fcc9e8f2068d8322f3d8dd0620000edd9bb7c9fe243\": rpc error: code = NotFound desc = could not find container \"231818c82ca9a23c34f94fcc9e8f2068d8322f3d8dd0620000edd9bb7c9fe243\": container with ID starting with 231818c82ca9a23c34f94fcc9e8f2068d8322f3d8dd0620000edd9bb7c9fe243 not found: ID does not exist"
May 25 13:25:49.850022 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.849992    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"]
May 25 13:25:49.853001 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.852971    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-08c99-predictor-559bfc6b95-vpb62"]
May 25 13:25:49.886636 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.886597    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdnnx\" (UniqueName: \"kubernetes.io/projected/d968d149-a207-4af3-91bf-db1ae7d8b745-kube-api-access-pdnnx\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:25:49.886636 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.886632    2569 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-08c99-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d968d149-a207-4af3-91bf-db1ae7d8b745-error-404-isvc-08c99-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:25:49.886814 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:49.886651    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d968d149-a207-4af3-91bf-db1ae7d8b745-proxy-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:25:50.597161 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:50.597123    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d968d149-a207-4af3-91bf-db1ae7d8b745" path="/var/lib/kubelet/pods/d968d149-a207-4af3-91bf-db1ae7d8b745/volumes"
May 25 13:25:53.127472 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:53.127427    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d" podUID="c713f40d-97e5-4aef-9cde-11fcce6f1ec4" containerName="sequence-graph-08c99" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:25:53.711287 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:53.711248    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8" podUID="de4f786e-ee21-4acf-a660-ec364cf8ac74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 13:25:54.830374 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:54.830344    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"
May 25 13:25:54.830881 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:54.830847    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb" podUID="ce196080-f42b-4918-b6f6-f42da3537e67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 25 13:25:58.127768 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:58.127722    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d" podUID="c713f40d-97e5-4aef-9cde-11fcce6f1ec4" containerName="sequence-graph-08c99" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:25:58.128189 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:25:58.127835    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d"
May 25 13:26:03.128356 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:03.128310    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d" podUID="c713f40d-97e5-4aef-9cde-11fcce6f1ec4" containerName="sequence-graph-08c99" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:26:03.711538 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:03.711496    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8" podUID="de4f786e-ee21-4acf-a660-ec364cf8ac74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 13:26:04.830876 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:04.830831    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb" podUID="ce196080-f42b-4918-b6f6-f42da3537e67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 25 13:26:08.127655 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:08.127618    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d" podUID="c713f40d-97e5-4aef-9cde-11fcce6f1ec4" containerName="sequence-graph-08c99" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:26:13.128860 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:13.128818    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d" podUID="c713f40d-97e5-4aef-9cde-11fcce6f1ec4" containerName="sequence-graph-08c99" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:26:13.711600 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:13.711566    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"
May 25 13:26:14.831022 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:14.830977    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb" podUID="ce196080-f42b-4918-b6f6-f42da3537e67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 25 13:26:15.895053 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:15.895027    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d"
May 25 13:26:15.913652 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:15.913621    2569 generic.go:358] "Generic (PLEG): container finished" podID="c713f40d-97e5-4aef-9cde-11fcce6f1ec4" containerID="fe1f8cbf0c3788995178478b0c4613691fcffb4716b452841e88a6a7c434fd24" exitCode=0
May 25 13:26:15.913814 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:15.913682    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d"
May 25 13:26:15.913814 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:15.913692    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d" event={"ID":"c713f40d-97e5-4aef-9cde-11fcce6f1ec4","Type":"ContainerDied","Data":"fe1f8cbf0c3788995178478b0c4613691fcffb4716b452841e88a6a7c434fd24"}
May 25 13:26:15.913814 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:15.913732    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d" event={"ID":"c713f40d-97e5-4aef-9cde-11fcce6f1ec4","Type":"ContainerDied","Data":"4cb87a8c759478161ce0f8a54ac4e3a51565ae80d27d9c8f8e93298bccecf2b2"}
May 25 13:26:15.913814 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:15.913748    2569 scope.go:117] "RemoveContainer" containerID="fe1f8cbf0c3788995178478b0c4613691fcffb4716b452841e88a6a7c434fd24"
May 25 13:26:15.923666 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:15.923646    2569 scope.go:117] "RemoveContainer" containerID="fe1f8cbf0c3788995178478b0c4613691fcffb4716b452841e88a6a7c434fd24"
May 25 13:26:15.924029 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:26:15.924001    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe1f8cbf0c3788995178478b0c4613691fcffb4716b452841e88a6a7c434fd24\": container with ID starting with fe1f8cbf0c3788995178478b0c4613691fcffb4716b452841e88a6a7c434fd24 not found: ID does not exist" containerID="fe1f8cbf0c3788995178478b0c4613691fcffb4716b452841e88a6a7c434fd24"
May 25 13:26:15.924100 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:15.924044    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe1f8cbf0c3788995178478b0c4613691fcffb4716b452841e88a6a7c434fd24"} err="failed to get container status \"fe1f8cbf0c3788995178478b0c4613691fcffb4716b452841e88a6a7c434fd24\": rpc error: code = NotFound desc = could not find container \"fe1f8cbf0c3788995178478b0c4613691fcffb4716b452841e88a6a7c434fd24\": container with ID starting with fe1f8cbf0c3788995178478b0c4613691fcffb4716b452841e88a6a7c434fd24 not found: ID does not exist"
May 25 13:26:15.941219 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:15.941183    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t"]
May 25 13:26:15.941623 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:15.941605    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c713f40d-97e5-4aef-9cde-11fcce6f1ec4" containerName="sequence-graph-08c99"
May 25 13:26:15.941720 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:15.941625    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c713f40d-97e5-4aef-9cde-11fcce6f1ec4" containerName="sequence-graph-08c99"
May 25 13:26:15.941720 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:15.941644    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d968d149-a207-4af3-91bf-db1ae7d8b745" containerName="kserve-container"
May 25 13:26:15.941720 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:15.941652    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d968d149-a207-4af3-91bf-db1ae7d8b745" containerName="kserve-container"
May 25 13:26:15.941720 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:15.941662    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d968d149-a207-4af3-91bf-db1ae7d8b745" containerName="kube-rbac-proxy"
May 25 13:26:15.941720 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:15.941672    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d968d149-a207-4af3-91bf-db1ae7d8b745" containerName="kube-rbac-proxy"
May 25 13:26:15.941945 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:15.941757    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="d968d149-a207-4af3-91bf-db1ae7d8b745" containerName="kserve-container"
May 25 13:26:15.941945 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:15.941774    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="d968d149-a207-4af3-91bf-db1ae7d8b745" containerName="kube-rbac-proxy"
May 25 13:26:15.941945 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:15.941788    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="c713f40d-97e5-4aef-9cde-11fcce6f1ec4" containerName="sequence-graph-08c99"
May 25 13:26:15.946212 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:15.946189    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t"
May 25 13:26:15.948618 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:15.948598    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-db114-serving-cert\""
May 25 13:26:15.948724 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:15.948668    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-db114-kube-rbac-proxy-sar-config\""
May 25 13:26:15.953179 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:15.953128    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t"]
May 25 13:26:16.010153 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:16.010038    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c713f40d-97e5-4aef-9cde-11fcce6f1ec4-openshift-service-ca-bundle\") pod \"c713f40d-97e5-4aef-9cde-11fcce6f1ec4\" (UID: \"c713f40d-97e5-4aef-9cde-11fcce6f1ec4\") "
May 25 13:26:16.010153 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:16.010087    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c713f40d-97e5-4aef-9cde-11fcce6f1ec4-proxy-tls\") pod \"c713f40d-97e5-4aef-9cde-11fcce6f1ec4\" (UID: \"c713f40d-97e5-4aef-9cde-11fcce6f1ec4\") "
May 25 13:26:16.010778 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:16.010748    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c713f40d-97e5-4aef-9cde-11fcce6f1ec4-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c713f40d-97e5-4aef-9cde-11fcce6f1ec4" (UID: "c713f40d-97e5-4aef-9cde-11fcce6f1ec4"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:26:16.012537 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:16.012518    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c713f40d-97e5-4aef-9cde-11fcce6f1ec4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c713f40d-97e5-4aef-9cde-11fcce6f1ec4" (UID: "c713f40d-97e5-4aef-9cde-11fcce6f1ec4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:26:16.110959 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:16.110916    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bb4be1b-7d3a-4c98-bc44-f1453b52244d-openshift-service-ca-bundle\") pod \"ensemble-graph-db114-85585bdbb4-6kg6t\" (UID: \"9bb4be1b-7d3a-4c98-bc44-f1453b52244d\") " pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t"
May 25 13:26:16.111139 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:16.110979    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bb4be1b-7d3a-4c98-bc44-f1453b52244d-proxy-tls\") pod \"ensemble-graph-db114-85585bdbb4-6kg6t\" (UID: \"9bb4be1b-7d3a-4c98-bc44-f1453b52244d\") " pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t"
May 25 13:26:16.111139 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:16.111075    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c713f40d-97e5-4aef-9cde-11fcce6f1ec4-openshift-service-ca-bundle\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:26:16.111139 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:16.111099    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c713f40d-97e5-4aef-9cde-11fcce6f1ec4-proxy-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:26:16.212132 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:16.212092    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bb4be1b-7d3a-4c98-bc44-f1453b52244d-openshift-service-ca-bundle\") pod \"ensemble-graph-db114-85585bdbb4-6kg6t\" (UID: \"9bb4be1b-7d3a-4c98-bc44-f1453b52244d\") " pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t"
May 25 13:26:16.212450 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:16.212429    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bb4be1b-7d3a-4c98-bc44-f1453b52244d-proxy-tls\") pod \"ensemble-graph-db114-85585bdbb4-6kg6t\" (UID: \"9bb4be1b-7d3a-4c98-bc44-f1453b52244d\") " pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t"
May 25 13:26:16.217073 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:16.215666    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bb4be1b-7d3a-4c98-bc44-f1453b52244d-proxy-tls\") pod \"ensemble-graph-db114-85585bdbb4-6kg6t\" (UID: \"9bb4be1b-7d3a-4c98-bc44-f1453b52244d\") " pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t"
May 25 13:26:16.217073 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:16.216341    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bb4be1b-7d3a-4c98-bc44-f1453b52244d-openshift-service-ca-bundle\") pod \"ensemble-graph-db114-85585bdbb4-6kg6t\" (UID: \"9bb4be1b-7d3a-4c98-bc44-f1453b52244d\") " pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t"
May 25 13:26:16.238706 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:16.238669    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d"]
May 25 13:26:16.242392 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:16.242366    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-08c99-84bc9f87db-h8b2d"]
May 25 13:26:16.258017 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:16.257990    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t"
May 25 13:26:16.390620 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:16.390574    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t"]
May 25 13:26:16.394217 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:26:16.394183    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb4be1b_7d3a_4c98_bc44_f1453b52244d.slice/crio-4e6f03e699e0c9301dc83f33a6c6eabe290b5ea4d6a61ff982ab4791ec75535f WatchSource:0}: Error finding container 4e6f03e699e0c9301dc83f33a6c6eabe290b5ea4d6a61ff982ab4791ec75535f: Status 404 returned error can't find the container with id 4e6f03e699e0c9301dc83f33a6c6eabe290b5ea4d6a61ff982ab4791ec75535f
May 25 13:26:16.597362 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:16.597333    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c713f40d-97e5-4aef-9cde-11fcce6f1ec4" path="/var/lib/kubelet/pods/c713f40d-97e5-4aef-9cde-11fcce6f1ec4/volumes"
May 25 13:26:16.918647 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:16.918546    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t" event={"ID":"9bb4be1b-7d3a-4c98-bc44-f1453b52244d","Type":"ContainerStarted","Data":"3f0a7b069d21f32ae52ab14a195ef843476c2d3100f94e69b6f5f7d7f4797f54"}
May 25 13:26:16.918647 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:16.918588    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t" event={"ID":"9bb4be1b-7d3a-4c98-bc44-f1453b52244d","Type":"ContainerStarted","Data":"4e6f03e699e0c9301dc83f33a6c6eabe290b5ea4d6a61ff982ab4791ec75535f"}
May 25 13:26:16.918647 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:16.918608    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t"
May 25 13:26:16.935130 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:16.935072    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t" podStartSLOduration=1.935053936 podStartE2EDuration="1.935053936s" podCreationTimestamp="2026-05-25 13:26:15 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:26:16.933995139 +0000 UTC m=+1292.936736896" watchObservedRunningTime="2026-05-25 13:26:16.935053936 +0000 UTC m=+1292.937795690"
May 25 13:26:22.930095 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:22.930064    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t"
May 25 13:26:24.831033 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:24.830992    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb" podUID="ce196080-f42b-4918-b6f6-f42da3537e67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 25 13:26:26.037982 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.037880    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t"]
May 25 13:26:26.038456 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.038147    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t" podUID="9bb4be1b-7d3a-4c98-bc44-f1453b52244d" containerName="ensemble-graph-db114" containerID="cri-o://3f0a7b069d21f32ae52ab14a195ef843476c2d3100f94e69b6f5f7d7f4797f54" gracePeriod=30
May 25 13:26:26.225691 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.225656    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"]
May 25 13:26:26.225994 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.225946    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8" podUID="de4f786e-ee21-4acf-a660-ec364cf8ac74" containerName="kserve-container" containerID="cri-o://4ba6a44912e6f4d0dd1f373c486eb2b2f2c265e7e5b78a40959b2498b907a139" gracePeriod=30
May 25 13:26:26.226087 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.225971    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8" podUID="de4f786e-ee21-4acf-a660-ec364cf8ac74" containerName="kube-rbac-proxy" containerID="cri-o://3c7856ab965df4f41796ba86e78e13fd50bc154fabc04cd72bba90b045532989" gracePeriod=30
May 25 13:26:26.298809 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.298716    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"]
May 25 13:26:26.303497 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.303473    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"
May 25 13:26:26.305994 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.305971    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3e647-kube-rbac-proxy-sar-config\""
May 25 13:26:26.306095 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.306015    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3e647-predictor-serving-cert\""
May 25 13:26:26.314765 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.314734    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"]
May 25 13:26:26.399948 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.399912    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3-proxy-tls\") pod \"error-404-isvc-3e647-predictor-9899bf78c-4dxp6\" (UID: \"09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3\") " pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"
May 25 13:26:26.400182 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.399969    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzmxg\" (UniqueName: \"kubernetes.io/projected/09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3-kube-api-access-pzmxg\") pod \"error-404-isvc-3e647-predictor-9899bf78c-4dxp6\" (UID: \"09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3\") " pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"
May 25 13:26:26.400182 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.399996    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-3e647-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3-error-404-isvc-3e647-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3e647-predictor-9899bf78c-4dxp6\" (UID: \"09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3\") " pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"
May 25 13:26:26.500685 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.500643    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3-proxy-tls\") pod \"error-404-isvc-3e647-predictor-9899bf78c-4dxp6\" (UID: \"09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3\") " pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"
May 25 13:26:26.500854 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.500708    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pzmxg\" (UniqueName: \"kubernetes.io/projected/09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3-kube-api-access-pzmxg\") pod \"error-404-isvc-3e647-predictor-9899bf78c-4dxp6\" (UID: \"09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3\") " pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"
May 25 13:26:26.500854 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.500746    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-3e647-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3-error-404-isvc-3e647-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3e647-predictor-9899bf78c-4dxp6\" (UID: \"09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3\") " pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"
May 25 13:26:26.501411 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.501385    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-3e647-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3-error-404-isvc-3e647-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3e647-predictor-9899bf78c-4dxp6\" (UID: \"09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3\") " pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"
May 25 13:26:26.503043 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.503025    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3-proxy-tls\") pod \"error-404-isvc-3e647-predictor-9899bf78c-4dxp6\" (UID: \"09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3\") " pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"
May 25 13:26:26.509052 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.509032    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzmxg\" (UniqueName: \"kubernetes.io/projected/09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3-kube-api-access-pzmxg\") pod \"error-404-isvc-3e647-predictor-9899bf78c-4dxp6\" (UID: \"09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3\") " pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"
May 25 13:26:26.615856 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.615822    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"
May 25 13:26:26.752284 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.752225    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"]
May 25 13:26:26.760317 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:26:26.760288    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e86a6b_5c8f_4e71_b465_cbe3c22c9ca3.slice/crio-b65857942350b0d588127504cc7b9ee46f5377f0b5d057277c01c9b066ce360c WatchSource:0}: Error finding container b65857942350b0d588127504cc7b9ee46f5377f0b5d057277c01c9b066ce360c: Status 404 returned error can't find the container with id b65857942350b0d588127504cc7b9ee46f5377f0b5d057277c01c9b066ce360c
May 25 13:26:26.956724 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.956684    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6" event={"ID":"09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3","Type":"ContainerStarted","Data":"399246ec781a2a8195ba0f93d4345a619578d27c032ad42604f6778bce8ab3fc"}
May 25 13:26:26.956724 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.956731    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6" event={"ID":"09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3","Type":"ContainerStarted","Data":"4d94ce926ebde1145db97c1ee5741c12a6621bc2b7d7aac65af62750730237ca"}
May 25 13:26:26.956992 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.956745    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6" event={"ID":"09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3","Type":"ContainerStarted","Data":"b65857942350b0d588127504cc7b9ee46f5377f0b5d057277c01c9b066ce360c"}
May 25 13:26:26.956992 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.956875    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"
May 25 13:26:26.956992 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.956905    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"
May 25 13:26:26.958076 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.958050    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6" podUID="09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 25 13:26:26.958383 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.958363    2569 generic.go:358] "Generic (PLEG): container finished" podID="de4f786e-ee21-4acf-a660-ec364cf8ac74" containerID="3c7856ab965df4f41796ba86e78e13fd50bc154fabc04cd72bba90b045532989" exitCode=2
May 25 13:26:26.958448 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.958429    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8" event={"ID":"de4f786e-ee21-4acf-a660-ec364cf8ac74","Type":"ContainerDied","Data":"3c7856ab965df4f41796ba86e78e13fd50bc154fabc04cd72bba90b045532989"}
May 25 13:26:26.979759 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:26.979707    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6" podStartSLOduration=0.979690364 podStartE2EDuration="979.690364ms" podCreationTimestamp="2026-05-25 13:26:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:26:26.977645406 +0000 UTC m=+1302.980387155" watchObservedRunningTime="2026-05-25 13:26:26.979690364 +0000 UTC m=+1302.982432115"
May 25 13:26:27.928868 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:27.928819    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t" podUID="9bb4be1b-7d3a-4c98-bc44-f1453b52244d" containerName="ensemble-graph-db114" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:26:27.962338 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:27.962301    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6" podUID="09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 25 13:26:28.705874 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:28.705826    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8" podUID="de4f786e-ee21-4acf-a660-ec364cf8ac74" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
May 25 13:26:29.579148 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:29.579122    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"
May 25 13:26:29.734516 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:29.734421    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de4f786e-ee21-4acf-a660-ec364cf8ac74-proxy-tls\") pod \"de4f786e-ee21-4acf-a660-ec364cf8ac74\" (UID: \"de4f786e-ee21-4acf-a660-ec364cf8ac74\") "
May 25 13:26:29.734516 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:29.734503    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-db114-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de4f786e-ee21-4acf-a660-ec364cf8ac74-error-404-isvc-db114-kube-rbac-proxy-sar-config\") pod \"de4f786e-ee21-4acf-a660-ec364cf8ac74\" (UID: \"de4f786e-ee21-4acf-a660-ec364cf8ac74\") "
May 25 13:26:29.734738 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:29.734538    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hp4gt\" (UniqueName: \"kubernetes.io/projected/de4f786e-ee21-4acf-a660-ec364cf8ac74-kube-api-access-hp4gt\") pod \"de4f786e-ee21-4acf-a660-ec364cf8ac74\" (UID: \"de4f786e-ee21-4acf-a660-ec364cf8ac74\") "
May 25 13:26:29.734853 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:29.734828    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de4f786e-ee21-4acf-a660-ec364cf8ac74-error-404-isvc-db114-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-db114-kube-rbac-proxy-sar-config") pod "de4f786e-ee21-4acf-a660-ec364cf8ac74" (UID: "de4f786e-ee21-4acf-a660-ec364cf8ac74"). InnerVolumeSpecName "error-404-isvc-db114-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:26:29.736577 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:29.736556    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de4f786e-ee21-4acf-a660-ec364cf8ac74-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "de4f786e-ee21-4acf-a660-ec364cf8ac74" (UID: "de4f786e-ee21-4acf-a660-ec364cf8ac74"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:26:29.736651 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:29.736627    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4f786e-ee21-4acf-a660-ec364cf8ac74-kube-api-access-hp4gt" (OuterVolumeSpecName: "kube-api-access-hp4gt") pod "de4f786e-ee21-4acf-a660-ec364cf8ac74" (UID: "de4f786e-ee21-4acf-a660-ec364cf8ac74"). InnerVolumeSpecName "kube-api-access-hp4gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:26:29.835834 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:29.835796    2569 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-db114-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de4f786e-ee21-4acf-a660-ec364cf8ac74-error-404-isvc-db114-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:26:29.835834 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:29.835835    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp4gt\" (UniqueName: \"kubernetes.io/projected/de4f786e-ee21-4acf-a660-ec364cf8ac74-kube-api-access-hp4gt\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:26:29.836037 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:29.835850    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de4f786e-ee21-4acf-a660-ec364cf8ac74-proxy-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:26:29.970077 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:29.970036    2569 generic.go:358] "Generic (PLEG): container finished" podID="de4f786e-ee21-4acf-a660-ec364cf8ac74" containerID="4ba6a44912e6f4d0dd1f373c486eb2b2f2c265e7e5b78a40959b2498b907a139" exitCode=0
May 25 13:26:29.970254 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:29.970116    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"
May 25 13:26:29.970254 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:29.970118    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8" event={"ID":"de4f786e-ee21-4acf-a660-ec364cf8ac74","Type":"ContainerDied","Data":"4ba6a44912e6f4d0dd1f373c486eb2b2f2c265e7e5b78a40959b2498b907a139"}
May 25 13:26:29.970254 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:29.970160    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8" event={"ID":"de4f786e-ee21-4acf-a660-ec364cf8ac74","Type":"ContainerDied","Data":"c67e0d8b528c3c97fbb00bfd356286ca55bb1994289c037dccfa8302c1795956"}
May 25 13:26:29.970254 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:29.970194    2569 scope.go:117] "RemoveContainer" containerID="3c7856ab965df4f41796ba86e78e13fd50bc154fabc04cd72bba90b045532989"
May 25 13:26:29.979427 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:29.979406    2569 scope.go:117] "RemoveContainer" containerID="4ba6a44912e6f4d0dd1f373c486eb2b2f2c265e7e5b78a40959b2498b907a139"
May 25 13:26:29.986856 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:29.986840    2569 scope.go:117] "RemoveContainer" containerID="3c7856ab965df4f41796ba86e78e13fd50bc154fabc04cd72bba90b045532989"
May 25 13:26:29.987094 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:26:29.987078    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c7856ab965df4f41796ba86e78e13fd50bc154fabc04cd72bba90b045532989\": container with ID starting with 3c7856ab965df4f41796ba86e78e13fd50bc154fabc04cd72bba90b045532989 not found: ID does not exist" containerID="3c7856ab965df4f41796ba86e78e13fd50bc154fabc04cd72bba90b045532989"
May 25 13:26:29.987134 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:29.987103    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c7856ab965df4f41796ba86e78e13fd50bc154fabc04cd72bba90b045532989"} err="failed to get container status \"3c7856ab965df4f41796ba86e78e13fd50bc154fabc04cd72bba90b045532989\": rpc error: code = NotFound desc = could not find container \"3c7856ab965df4f41796ba86e78e13fd50bc154fabc04cd72bba90b045532989\": container with ID starting with 3c7856ab965df4f41796ba86e78e13fd50bc154fabc04cd72bba90b045532989 not found: ID does not exist"
May 25 13:26:29.987134 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:29.987120    2569 scope.go:117] "RemoveContainer" containerID="4ba6a44912e6f4d0dd1f373c486eb2b2f2c265e7e5b78a40959b2498b907a139"
May 25 13:26:29.987356 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:26:29.987334    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba6a44912e6f4d0dd1f373c486eb2b2f2c265e7e5b78a40959b2498b907a139\": container with ID starting with 4ba6a44912e6f4d0dd1f373c486eb2b2f2c265e7e5b78a40959b2498b907a139 not found: ID does not exist" containerID="4ba6a44912e6f4d0dd1f373c486eb2b2f2c265e7e5b78a40959b2498b907a139"
May 25 13:26:29.987405 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:29.987360    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba6a44912e6f4d0dd1f373c486eb2b2f2c265e7e5b78a40959b2498b907a139"} err="failed to get container status \"4ba6a44912e6f4d0dd1f373c486eb2b2f2c265e7e5b78a40959b2498b907a139\": rpc error: code = NotFound desc = could not find container \"4ba6a44912e6f4d0dd1f373c486eb2b2f2c265e7e5b78a40959b2498b907a139\": container with ID starting with 4ba6a44912e6f4d0dd1f373c486eb2b2f2c265e7e5b78a40959b2498b907a139 not found: ID does not exist"
May 25 13:26:29.991811 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:29.991784    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"]
May 25 13:26:29.994343 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:29.994322    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db114-predictor-7d7bc84957-tkjp8"]
May 25 13:26:30.596791 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:30.596757    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de4f786e-ee21-4acf-a660-ec364cf8ac74" path="/var/lib/kubelet/pods/de4f786e-ee21-4acf-a660-ec364cf8ac74/volumes"
May 25 13:26:32.929199 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:32.929138    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t" podUID="9bb4be1b-7d3a-4c98-bc44-f1453b52244d" containerName="ensemble-graph-db114" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:26:32.967083 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:32.967057    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"
May 25 13:26:32.967536 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:32.967511    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6" podUID="09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 25 13:26:34.832163 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:34.832134    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"
May 25 13:26:37.928063 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:37.928028    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t" podUID="9bb4be1b-7d3a-4c98-bc44-f1453b52244d" containerName="ensemble-graph-db114" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:26:37.928549 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:37.928127    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t"
May 25 13:26:42.928270 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:42.928231    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t" podUID="9bb4be1b-7d3a-4c98-bc44-f1453b52244d" containerName="ensemble-graph-db114" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:26:42.967895 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:42.967857    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6" podUID="09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 25 13:26:45.914971 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:45.914939    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw"]
May 25 13:26:45.915393 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:45.915350    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de4f786e-ee21-4acf-a660-ec364cf8ac74" containerName="kserve-container"
May 25 13:26:45.915393 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:45.915366    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="de4f786e-ee21-4acf-a660-ec364cf8ac74" containerName="kserve-container"
May 25 13:26:45.915393 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:45.915381    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de4f786e-ee21-4acf-a660-ec364cf8ac74" containerName="kube-rbac-proxy"
May 25 13:26:45.915393 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:45.915387    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="de4f786e-ee21-4acf-a660-ec364cf8ac74" containerName="kube-rbac-proxy"
May 25 13:26:45.915519 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:45.915445    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="de4f786e-ee21-4acf-a660-ec364cf8ac74" containerName="kube-rbac-proxy"
May 25 13:26:45.915519 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:45.915459    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="de4f786e-ee21-4acf-a660-ec364cf8ac74" containerName="kserve-container"
May 25 13:26:45.919792 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:45.919773    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw"
May 25 13:26:45.921974 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:45.921956    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-9ee57-serving-cert\""
May 25 13:26:45.922065 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:45.921961    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-9ee57-kube-rbac-proxy-sar-config\""
May 25 13:26:45.925134 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:45.925109    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw"]
May 25 13:26:45.969302 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:45.969260    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7340801-9a36-4fe1-be78-44e58bb71d75-openshift-service-ca-bundle\") pod \"sequence-graph-9ee57-67b68b489d-bvwfw\" (UID: \"b7340801-9a36-4fe1-be78-44e58bb71d75\") " pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw"
May 25 13:26:45.969479 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:45.969314    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7340801-9a36-4fe1-be78-44e58bb71d75-proxy-tls\") pod \"sequence-graph-9ee57-67b68b489d-bvwfw\" (UID: \"b7340801-9a36-4fe1-be78-44e58bb71d75\") " pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw"
May 25 13:26:46.070355 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:46.070317    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7340801-9a36-4fe1-be78-44e58bb71d75-proxy-tls\") pod \"sequence-graph-9ee57-67b68b489d-bvwfw\" (UID: \"b7340801-9a36-4fe1-be78-44e58bb71d75\") " pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw"
May 25 13:26:46.070505 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:46.070430    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7340801-9a36-4fe1-be78-44e58bb71d75-openshift-service-ca-bundle\") pod \"sequence-graph-9ee57-67b68b489d-bvwfw\" (UID: \"b7340801-9a36-4fe1-be78-44e58bb71d75\") " pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw"
May 25 13:26:46.070505 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:26:46.070475    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-9ee57-serving-cert: secret "sequence-graph-9ee57-serving-cert" not found
May 25 13:26:46.070597 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:26:46.070562    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7340801-9a36-4fe1-be78-44e58bb71d75-proxy-tls podName:b7340801-9a36-4fe1-be78-44e58bb71d75 nodeName:}" failed. No retries permitted until 2026-05-25 13:26:46.570541651 +0000 UTC m=+1322.573283403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b7340801-9a36-4fe1-be78-44e58bb71d75-proxy-tls") pod "sequence-graph-9ee57-67b68b489d-bvwfw" (UID: "b7340801-9a36-4fe1-be78-44e58bb71d75") : secret "sequence-graph-9ee57-serving-cert" not found
May 25 13:26:46.071107 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:46.071088    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7340801-9a36-4fe1-be78-44e58bb71d75-openshift-service-ca-bundle\") pod \"sequence-graph-9ee57-67b68b489d-bvwfw\" (UID: \"b7340801-9a36-4fe1-be78-44e58bb71d75\") " pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw"
May 25 13:26:46.575783 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:46.575737    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7340801-9a36-4fe1-be78-44e58bb71d75-proxy-tls\") pod \"sequence-graph-9ee57-67b68b489d-bvwfw\" (UID: \"b7340801-9a36-4fe1-be78-44e58bb71d75\") " pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw"
May 25 13:26:46.578096 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:46.578076    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7340801-9a36-4fe1-be78-44e58bb71d75-proxy-tls\") pod \"sequence-graph-9ee57-67b68b489d-bvwfw\" (UID: \"b7340801-9a36-4fe1-be78-44e58bb71d75\") " pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw"
May 25 13:26:46.833868 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:46.833785    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw"
May 25 13:26:46.964214 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:46.964189    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw"]
May 25 13:26:47.033117 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:47.033082    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw" event={"ID":"b7340801-9a36-4fe1-be78-44e58bb71d75","Type":"ContainerStarted","Data":"5d4e47856025e9011e1d3d3e226018cbf7e380bc0b810cf1643205926b7591da"}
May 25 13:26:47.033268 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:47.033121    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw" event={"ID":"b7340801-9a36-4fe1-be78-44e58bb71d75","Type":"ContainerStarted","Data":"d27667aedbab4d66832bc6048db3bd1a813799d5df7dc01e2718a806928d04df"}
May 25 13:26:47.033268 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:47.033216    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw"
May 25 13:26:47.053116 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:47.053069    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw" podStartSLOduration=2.053054673 podStartE2EDuration="2.053054673s" podCreationTimestamp="2026-05-25 13:26:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:26:47.051074321 +0000 UTC m=+1323.053816076" watchObservedRunningTime="2026-05-25 13:26:47.053054673 +0000 UTC m=+1323.055796426"
May 25 13:26:47.927748 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:47.927712    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t" podUID="9bb4be1b-7d3a-4c98-bc44-f1453b52244d" containerName="ensemble-graph-db114" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:26:52.927999 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:52.927959    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t" podUID="9bb4be1b-7d3a-4c98-bc44-f1453b52244d" containerName="ensemble-graph-db114" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:26:52.967885 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:52.967844    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6" podUID="09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 25 13:26:53.049934 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:53.049896    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw"
May 25 13:26:56.009694 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.009660    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw"]
May 25 13:26:56.010326 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.009962    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw" podUID="b7340801-9a36-4fe1-be78-44e58bb71d75" containerName="sequence-graph-9ee57" containerID="cri-o://5d4e47856025e9011e1d3d3e226018cbf7e380bc0b810cf1643205926b7591da" gracePeriod=30
May 25 13:26:56.072838 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.072720    2569 generic.go:358] "Generic (PLEG): container finished" podID="9bb4be1b-7d3a-4c98-bc44-f1453b52244d" containerID="3f0a7b069d21f32ae52ab14a195ef843476c2d3100f94e69b6f5f7d7f4797f54" exitCode=0
May 25 13:26:56.072838 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.072788    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t" event={"ID":"9bb4be1b-7d3a-4c98-bc44-f1453b52244d","Type":"ContainerDied","Data":"3f0a7b069d21f32ae52ab14a195ef843476c2d3100f94e69b6f5f7d7f4797f54"}
May 25 13:26:56.084545 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.084502    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"]
May 25 13:26:56.084928 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.084900    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb" podUID="ce196080-f42b-4918-b6f6-f42da3537e67" containerName="kserve-container" containerID="cri-o://88a6c18d2e11e2450940016d5d8af3f158c5e7f1089602d9399f6bc83df99fd4" gracePeriod=30
May 25 13:26:56.085029 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.084977    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb" podUID="ce196080-f42b-4918-b6f6-f42da3537e67" containerName="kube-rbac-proxy" containerID="cri-o://f22e6dd75d905aa5162b4e376ed698e2e910c2f108542f31572436547472c40a" gracePeriod=30
May 25 13:26:56.215105 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.215080    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t"
May 25 13:26:56.255799 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.255771    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bb4be1b-7d3a-4c98-bc44-f1453b52244d-proxy-tls\") pod \"9bb4be1b-7d3a-4c98-bc44-f1453b52244d\" (UID: \"9bb4be1b-7d3a-4c98-bc44-f1453b52244d\") "
May 25 13:26:56.255799 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.255812    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bb4be1b-7d3a-4c98-bc44-f1453b52244d-openshift-service-ca-bundle\") pod \"9bb4be1b-7d3a-4c98-bc44-f1453b52244d\" (UID: \"9bb4be1b-7d3a-4c98-bc44-f1453b52244d\") "
May 25 13:26:56.256298 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.256206    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb4be1b-7d3a-4c98-bc44-f1453b52244d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "9bb4be1b-7d3a-4c98-bc44-f1453b52244d" (UID: "9bb4be1b-7d3a-4c98-bc44-f1453b52244d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:26:56.258635 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.258584    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bb4be1b-7d3a-4c98-bc44-f1453b52244d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9bb4be1b-7d3a-4c98-bc44-f1453b52244d" (UID: "9bb4be1b-7d3a-4c98-bc44-f1453b52244d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:26:56.311873 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.311834    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"]
May 25 13:26:56.312363 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.312348    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bb4be1b-7d3a-4c98-bc44-f1453b52244d" containerName="ensemble-graph-db114"
May 25 13:26:56.312422 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.312367    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bb4be1b-7d3a-4c98-bc44-f1453b52244d" containerName="ensemble-graph-db114"
May 25 13:26:56.312481 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.312470    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="9bb4be1b-7d3a-4c98-bc44-f1453b52244d" containerName="ensemble-graph-db114"
May 25 13:26:56.316119 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.316097    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"
May 25 13:26:56.318339 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.318315    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-72c4b-predictor-serving-cert\""
May 25 13:26:56.318460 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.318345    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-72c4b-kube-rbac-proxy-sar-config\""
May 25 13:26:56.323972 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.323446    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"]
May 25 13:26:56.357709 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.357673    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f074234f-c787-4e18-ab9c-3cf3e375eecb-proxy-tls\") pod \"error-404-isvc-72c4b-predictor-549bf86d85-lbvpt\" (UID: \"f074234f-c787-4e18-ab9c-3cf3e375eecb\") " pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"
May 25 13:26:56.357859 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.357760    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-72c4b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f074234f-c787-4e18-ab9c-3cf3e375eecb-error-404-isvc-72c4b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-72c4b-predictor-549bf86d85-lbvpt\" (UID: \"f074234f-c787-4e18-ab9c-3cf3e375eecb\") " pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"
May 25 13:26:56.357859 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.357803    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84nnw\" (UniqueName: \"kubernetes.io/projected/f074234f-c787-4e18-ab9c-3cf3e375eecb-kube-api-access-84nnw\") pod \"error-404-isvc-72c4b-predictor-549bf86d85-lbvpt\" (UID: \"f074234f-c787-4e18-ab9c-3cf3e375eecb\") " pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"
May 25 13:26:56.357859 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.357853    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bb4be1b-7d3a-4c98-bc44-f1453b52244d-proxy-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:26:56.358101 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.357870    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bb4be1b-7d3a-4c98-bc44-f1453b52244d-openshift-service-ca-bundle\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:26:56.458427 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.458375    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-84nnw\" (UniqueName: \"kubernetes.io/projected/f074234f-c787-4e18-ab9c-3cf3e375eecb-kube-api-access-84nnw\") pod \"error-404-isvc-72c4b-predictor-549bf86d85-lbvpt\" (UID: \"f074234f-c787-4e18-ab9c-3cf3e375eecb\") " pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"
May 25 13:26:56.458619 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.458473    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f074234f-c787-4e18-ab9c-3cf3e375eecb-proxy-tls\") pod \"error-404-isvc-72c4b-predictor-549bf86d85-lbvpt\" (UID: \"f074234f-c787-4e18-ab9c-3cf3e375eecb\") " pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"
May 25 13:26:56.458619 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.458543    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-72c4b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f074234f-c787-4e18-ab9c-3cf3e375eecb-error-404-isvc-72c4b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-72c4b-predictor-549bf86d85-lbvpt\" (UID: \"f074234f-c787-4e18-ab9c-3cf3e375eecb\") " pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"
May 25 13:26:56.459157 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.459128    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-72c4b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f074234f-c787-4e18-ab9c-3cf3e375eecb-error-404-isvc-72c4b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-72c4b-predictor-549bf86d85-lbvpt\" (UID: \"f074234f-c787-4e18-ab9c-3cf3e375eecb\") " pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"
May 25 13:26:56.461038 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.461017    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f074234f-c787-4e18-ab9c-3cf3e375eecb-proxy-tls\") pod \"error-404-isvc-72c4b-predictor-549bf86d85-lbvpt\" (UID: \"f074234f-c787-4e18-ab9c-3cf3e375eecb\") " pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"
May 25 13:26:56.467723 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.467698    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84nnw\" (UniqueName: \"kubernetes.io/projected/f074234f-c787-4e18-ab9c-3cf3e375eecb-kube-api-access-84nnw\") pod \"error-404-isvc-72c4b-predictor-549bf86d85-lbvpt\" (UID: \"f074234f-c787-4e18-ab9c-3cf3e375eecb\") " pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"
May 25 13:26:56.630810 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.630778    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"
May 25 13:26:56.762804 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:56.762771    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"]
May 25 13:26:57.077923 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:57.077894    2569 generic.go:358] "Generic (PLEG): container finished" podID="ce196080-f42b-4918-b6f6-f42da3537e67" containerID="f22e6dd75d905aa5162b4e376ed698e2e910c2f108542f31572436547472c40a" exitCode=2
May 25 13:26:57.078372 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:57.077958    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb" event={"ID":"ce196080-f42b-4918-b6f6-f42da3537e67","Type":"ContainerDied","Data":"f22e6dd75d905aa5162b4e376ed698e2e910c2f108542f31572436547472c40a"}
May 25 13:26:57.079273 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:57.079249    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t"
May 25 13:26:57.079273 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:57.079253    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t" event={"ID":"9bb4be1b-7d3a-4c98-bc44-f1453b52244d","Type":"ContainerDied","Data":"4e6f03e699e0c9301dc83f33a6c6eabe290b5ea4d6a61ff982ab4791ec75535f"}
May 25 13:26:57.079473 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:57.079302    2569 scope.go:117] "RemoveContainer" containerID="3f0a7b069d21f32ae52ab14a195ef843476c2d3100f94e69b6f5f7d7f4797f54"
May 25 13:26:57.080879 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:57.080856    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt" event={"ID":"f074234f-c787-4e18-ab9c-3cf3e375eecb","Type":"ContainerStarted","Data":"afda7a17a6650bbf9b4c5616f8b8ad20eae73a2cd8e4d1fb58e997b1a9624e1d"}
May 25 13:26:57.080989 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:57.080886    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt" event={"ID":"f074234f-c787-4e18-ab9c-3cf3e375eecb","Type":"ContainerStarted","Data":"0cbcb38cb3a58f051b3f6d61261a3089f865ed89323764a6fe98f27c7872e3d4"}
May 25 13:26:57.080989 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:57.080901    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt" event={"ID":"f074234f-c787-4e18-ab9c-3cf3e375eecb","Type":"ContainerStarted","Data":"3b59742a2e6977b80607396fc45a740c3b91886eec0ba4046db5189722f37cd7"}
May 25 13:26:57.081093 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:57.081034    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"
May 25 13:26:57.094653 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:57.094631    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t"]
May 25 13:26:57.098641 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:57.098621    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-db114-85585bdbb4-6kg6t"]
May 25 13:26:57.113783 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:57.113704    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt" podStartSLOduration=1.113691848 podStartE2EDuration="1.113691848s" podCreationTimestamp="2026-05-25 13:26:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:26:57.112396168 +0000 UTC m=+1333.115137920" watchObservedRunningTime="2026-05-25 13:26:57.113691848 +0000 UTC m=+1333.116433602"
May 25 13:26:58.046769 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:58.046731    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw" podUID="b7340801-9a36-4fe1-be78-44e58bb71d75" containerName="sequence-graph-9ee57" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:26:58.086260 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:58.086227    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"
May 25 13:26:58.087576 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:58.087546    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt" podUID="f074234f-c787-4e18-ab9c-3cf3e375eecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 13:26:58.597439 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:58.597404    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bb4be1b-7d3a-4c98-bc44-f1453b52244d" path="/var/lib/kubelet/pods/9bb4be1b-7d3a-4c98-bc44-f1453b52244d/volumes"
May 25 13:26:59.089105 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:59.089008    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt" podUID="f074234f-c787-4e18-ab9c-3cf3e375eecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 13:26:59.544125 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:59.544102    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"
May 25 13:26:59.586475 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:59.586433    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-9ee57-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce196080-f42b-4918-b6f6-f42da3537e67-error-404-isvc-9ee57-kube-rbac-proxy-sar-config\") pod \"ce196080-f42b-4918-b6f6-f42da3537e67\" (UID: \"ce196080-f42b-4918-b6f6-f42da3537e67\") "
May 25 13:26:59.586662 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:59.586642    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce196080-f42b-4918-b6f6-f42da3537e67-proxy-tls\") pod \"ce196080-f42b-4918-b6f6-f42da3537e67\" (UID: \"ce196080-f42b-4918-b6f6-f42da3537e67\") "
May 25 13:26:59.586758 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:59.586712    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzp98\" (UniqueName: \"kubernetes.io/projected/ce196080-f42b-4918-b6f6-f42da3537e67-kube-api-access-vzp98\") pod \"ce196080-f42b-4918-b6f6-f42da3537e67\" (UID: \"ce196080-f42b-4918-b6f6-f42da3537e67\") "
May 25 13:26:59.587017 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:59.586984    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce196080-f42b-4918-b6f6-f42da3537e67-error-404-isvc-9ee57-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-9ee57-kube-rbac-proxy-sar-config") pod "ce196080-f42b-4918-b6f6-f42da3537e67" (UID: "ce196080-f42b-4918-b6f6-f42da3537e67"). InnerVolumeSpecName "error-404-isvc-9ee57-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:26:59.588758 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:59.588727    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce196080-f42b-4918-b6f6-f42da3537e67-kube-api-access-vzp98" (OuterVolumeSpecName: "kube-api-access-vzp98") pod "ce196080-f42b-4918-b6f6-f42da3537e67" (UID: "ce196080-f42b-4918-b6f6-f42da3537e67"). InnerVolumeSpecName "kube-api-access-vzp98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:26:59.588854 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:59.588753    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce196080-f42b-4918-b6f6-f42da3537e67-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ce196080-f42b-4918-b6f6-f42da3537e67" (UID: "ce196080-f42b-4918-b6f6-f42da3537e67"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:26:59.687829 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:59.687749    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce196080-f42b-4918-b6f6-f42da3537e67-proxy-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:26:59.687829 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:59.687782    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzp98\" (UniqueName: \"kubernetes.io/projected/ce196080-f42b-4918-b6f6-f42da3537e67-kube-api-access-vzp98\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:26:59.687829 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:26:59.687794    2569 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-9ee57-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce196080-f42b-4918-b6f6-f42da3537e67-error-404-isvc-9ee57-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:27:00.093480 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:00.093374    2569 generic.go:358] "Generic (PLEG): container finished" podID="ce196080-f42b-4918-b6f6-f42da3537e67" containerID="88a6c18d2e11e2450940016d5d8af3f158c5e7f1089602d9399f6bc83df99fd4" exitCode=0
May 25 13:27:00.093480 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:00.093461    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb" event={"ID":"ce196080-f42b-4918-b6f6-f42da3537e67","Type":"ContainerDied","Data":"88a6c18d2e11e2450940016d5d8af3f158c5e7f1089602d9399f6bc83df99fd4"}
May 25 13:27:00.094022 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:00.093513    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb" event={"ID":"ce196080-f42b-4918-b6f6-f42da3537e67","Type":"ContainerDied","Data":"663e2f5315b1291867fd6ec4171cda10b879ddec45891b8b2852fb6094e6dd96"}
May 25 13:27:00.094022 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:00.093533    2569 scope.go:117] "RemoveContainer" containerID="f22e6dd75d905aa5162b4e376ed698e2e910c2f108542f31572436547472c40a"
May 25 13:27:00.094022 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:00.093478    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"
May 25 13:27:00.102638 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:00.102619    2569 scope.go:117] "RemoveContainer" containerID="88a6c18d2e11e2450940016d5d8af3f158c5e7f1089602d9399f6bc83df99fd4"
May 25 13:27:00.109857 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:00.109830    2569 scope.go:117] "RemoveContainer" containerID="f22e6dd75d905aa5162b4e376ed698e2e910c2f108542f31572436547472c40a"
May 25 13:27:00.110103 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:27:00.110086    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f22e6dd75d905aa5162b4e376ed698e2e910c2f108542f31572436547472c40a\": container with ID starting with f22e6dd75d905aa5162b4e376ed698e2e910c2f108542f31572436547472c40a not found: ID does not exist" containerID="f22e6dd75d905aa5162b4e376ed698e2e910c2f108542f31572436547472c40a"
May 25 13:27:00.110148 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:00.110112    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f22e6dd75d905aa5162b4e376ed698e2e910c2f108542f31572436547472c40a"} err="failed to get container status \"f22e6dd75d905aa5162b4e376ed698e2e910c2f108542f31572436547472c40a\": rpc error: code = NotFound desc = could not find container \"f22e6dd75d905aa5162b4e376ed698e2e910c2f108542f31572436547472c40a\": container with ID starting with f22e6dd75d905aa5162b4e376ed698e2e910c2f108542f31572436547472c40a not found: ID does not exist"
May 25 13:27:00.110148 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:00.110131    2569 scope.go:117] "RemoveContainer" containerID="88a6c18d2e11e2450940016d5d8af3f158c5e7f1089602d9399f6bc83df99fd4"
May 25 13:27:00.110414 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:27:00.110388    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88a6c18d2e11e2450940016d5d8af3f158c5e7f1089602d9399f6bc83df99fd4\": container with ID starting with 88a6c18d2e11e2450940016d5d8af3f158c5e7f1089602d9399f6bc83df99fd4 not found: ID does not exist" containerID="88a6c18d2e11e2450940016d5d8af3f158c5e7f1089602d9399f6bc83df99fd4"
May 25 13:27:00.110488 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:00.110421    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a6c18d2e11e2450940016d5d8af3f158c5e7f1089602d9399f6bc83df99fd4"} err="failed to get container status \"88a6c18d2e11e2450940016d5d8af3f158c5e7f1089602d9399f6bc83df99fd4\": rpc error: code = NotFound desc = could not find container \"88a6c18d2e11e2450940016d5d8af3f158c5e7f1089602d9399f6bc83df99fd4\": container with ID starting with 88a6c18d2e11e2450940016d5d8af3f158c5e7f1089602d9399f6bc83df99fd4 not found: ID does not exist"
May 25 13:27:00.114942 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:00.114919    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"]
May 25 13:27:00.121239 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:00.121217    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9ee57-predictor-d974ddf95-6jxwb"]
May 25 13:27:00.597714 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:00.597678    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce196080-f42b-4918-b6f6-f42da3537e67" path="/var/lib/kubelet/pods/ce196080-f42b-4918-b6f6-f42da3537e67/volumes"
May 25 13:27:02.967660 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:02.967623    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6" podUID="09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 25 13:27:03.046778 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:03.046741    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw" podUID="b7340801-9a36-4fe1-be78-44e58bb71d75" containerName="sequence-graph-9ee57" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:27:04.093372 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:04.093292    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"
May 25 13:27:04.093869 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:04.093840    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt" podUID="f074234f-c787-4e18-ab9c-3cf3e375eecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 13:27:08.046525 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:08.046482    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw" podUID="b7340801-9a36-4fe1-be78-44e58bb71d75" containerName="sequence-graph-9ee57" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:27:08.046960 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:08.046597    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw"
May 25 13:27:12.967646 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:12.967608    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6" podUID="09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 25 13:27:13.046908 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:13.046862    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw" podUID="b7340801-9a36-4fe1-be78-44e58bb71d75" containerName="sequence-graph-9ee57" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:27:14.094527 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:14.094482    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt" podUID="f074234f-c787-4e18-ab9c-3cf3e375eecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 13:27:16.438337 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:27:16.438305    2569 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/d41d0380cb199e53bffe03aefa54d0738ceedc0de6bdfe17fe4147408c219fdc/diff" to get inode usage: stat /var/lib/containers/storage/overlay/d41d0380cb199e53bffe03aefa54d0738ceedc0de6bdfe17fe4147408c219fdc/diff: no such file or directory, extraDiskErr: could not stat "/var/log/pods/kserve-ci-e2e-test_ensemble-graph-db114-85585bdbb4-6kg6t_9bb4be1b-7d3a-4c98-bc44-f1453b52244d/ensemble-graph-db114/0.log" to get inode usage: stat /var/log/pods/kserve-ci-e2e-test_ensemble-graph-db114-85585bdbb4-6kg6t_9bb4be1b-7d3a-4c98-bc44-f1453b52244d/ensemble-graph-db114/0.log: no such file or directory
May 25 13:27:18.047457 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:18.047417    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw" podUID="b7340801-9a36-4fe1-be78-44e58bb71d75" containerName="sequence-graph-9ee57" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:27:22.968313 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:22.968283    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"
May 25 13:27:23.046534 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:23.046494    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw" podUID="b7340801-9a36-4fe1-be78-44e58bb71d75" containerName="sequence-graph-9ee57" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:27:24.094752 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:24.094715    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt" podUID="f074234f-c787-4e18-ab9c-3cf3e375eecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 13:27:26.057153 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:27:26.057077    2569 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7340801_9a36_4fe1_be78_44e58bb71d75.slice/crio-5d4e47856025e9011e1d3d3e226018cbf7e380bc0b810cf1643205926b7591da.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb4be1b_7d3a_4c98_bc44_f1453b52244d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce196080_f42b_4918_b6f6_f42da3537e67.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce196080_f42b_4918_b6f6_f42da3537e67.slice/crio-conmon-88a6c18d2e11e2450940016d5d8af3f158c5e7f1089602d9399f6bc83df99fd4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb4be1b_7d3a_4c98_bc44_f1453b52244d.slice/crio-4e6f03e699e0c9301dc83f33a6c6eabe290b5ea4d6a61ff982ab4791ec75535f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce196080_f42b_4918_b6f6_f42da3537e67.slice/crio-conmon-f22e6dd75d905aa5162b4e376ed698e2e910c2f108542f31572436547472c40a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb4be1b_7d3a_4c98_bc44_f1453b52244d.slice/crio-3f0a7b069d21f32ae52ab14a195ef843476c2d3100f94e69b6f5f7d7f4797f54.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce196080_f42b_4918_b6f6_f42da3537e67.slice/crio-663e2f5315b1291867fd6ec4171cda10b879ddec45891b8b2852fb6094e6dd96\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7340801_9a36_4fe1_be78_44e58bb71d75.slice/crio-conmon-5d4e47856025e9011e1d3d3e226018cbf7e380bc0b810cf1643205926b7591da.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb4be1b_7d3a_4c98_bc44_f1453b52244d.slice/crio-conmon-3f0a7b069d21f32ae52ab14a195ef843476c2d3100f94e69b6f5f7d7f4797f54.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce196080_f42b_4918_b6f6_f42da3537e67.slice/crio-f22e6dd75d905aa5162b4e376ed698e2e910c2f108542f31572436547472c40a.scope\": RecentStats: unable to find data in memory cache]"
May 25 13:27:26.057623 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:27:26.057153    2569 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce196080_f42b_4918_b6f6_f42da3537e67.slice/crio-f22e6dd75d905aa5162b4e376ed698e2e910c2f108542f31572436547472c40a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb4be1b_7d3a_4c98_bc44_f1453b52244d.slice/crio-4e6f03e699e0c9301dc83f33a6c6eabe290b5ea4d6a61ff982ab4791ec75535f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7340801_9a36_4fe1_be78_44e58bb71d75.slice/crio-conmon-5d4e47856025e9011e1d3d3e226018cbf7e380bc0b810cf1643205926b7591da.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb4be1b_7d3a_4c98_bc44_f1453b52244d.slice/crio-conmon-3f0a7b069d21f32ae52ab14a195ef843476c2d3100f94e69b6f5f7d7f4797f54.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce196080_f42b_4918_b6f6_f42da3537e67.slice/crio-conmon-88a6c18d2e11e2450940016d5d8af3f158c5e7f1089602d9399f6bc83df99fd4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb4be1b_7d3a_4c98_bc44_f1453b52244d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce196080_f42b_4918_b6f6_f42da3537e67.slice/crio-conmon-f22e6dd75d905aa5162b4e376ed698e2e910c2f108542f31572436547472c40a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb4be1b_7d3a_4c98_bc44_f1453b52244d.slice/crio-3f0a7b069d21f32ae52ab14a195ef843476c2d3100f94e69b6f5f7d7f4797f54.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce196080_f42b_4918_b6f6_f42da3537e67.slice/crio-88a6c18d2e11e2450940016d5d8af3f158c5e7f1089602d9399f6bc83df99fd4.scope\": RecentStats: unable to find data in memory cache]"
May 25 13:27:26.057623 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:27:26.057163    2569 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce196080_f42b_4918_b6f6_f42da3537e67.slice/crio-f22e6dd75d905aa5162b4e376ed698e2e910c2f108542f31572436547472c40a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb4be1b_7d3a_4c98_bc44_f1453b52244d.slice/crio-conmon-3f0a7b069d21f32ae52ab14a195ef843476c2d3100f94e69b6f5f7d7f4797f54.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb4be1b_7d3a_4c98_bc44_f1453b52244d.slice/crio-3f0a7b069d21f32ae52ab14a195ef843476c2d3100f94e69b6f5f7d7f4797f54.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb4be1b_7d3a_4c98_bc44_f1453b52244d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce196080_f42b_4918_b6f6_f42da3537e67.slice/crio-conmon-f22e6dd75d905aa5162b4e376ed698e2e910c2f108542f31572436547472c40a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce196080_f42b_4918_b6f6_f42da3537e67.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb4be1b_7d3a_4c98_bc44_f1453b52244d.slice/crio-4e6f03e699e0c9301dc83f33a6c6eabe290b5ea4d6a61ff982ab4791ec75535f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce196080_f42b_4918_b6f6_f42da3537e67.slice/crio-88a6c18d2e11e2450940016d5d8af3f158c5e7f1089602d9399f6bc83df99fd4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7340801_9a36_4fe1_be78_44e58bb71d75.slice/crio-conmon-5d4e47856025e9011e1d3d3e226018cbf7e380bc0b810cf1643205926b7591da.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce196080_f42b_4918_b6f6_f42da3537e67.slice/crio-conmon-88a6c18d2e11e2450940016d5d8af3f158c5e7f1089602d9399f6bc83df99fd4.scope\": RecentStats: unable to find data in memory cache]"
May 25 13:27:26.182272 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.182234    2569 generic.go:358] "Generic (PLEG): container finished" podID="b7340801-9a36-4fe1-be78-44e58bb71d75" containerID="5d4e47856025e9011e1d3d3e226018cbf7e380bc0b810cf1643205926b7591da" exitCode=137
May 25 13:27:26.182389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.182290    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw" event={"ID":"b7340801-9a36-4fe1-be78-44e58bb71d75","Type":"ContainerDied","Data":"5d4e47856025e9011e1d3d3e226018cbf7e380bc0b810cf1643205926b7591da"}
May 25 13:27:26.197154 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.197126    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw"
May 25 13:27:26.222936 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.222900    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7340801-9a36-4fe1-be78-44e58bb71d75-proxy-tls\") pod \"b7340801-9a36-4fe1-be78-44e58bb71d75\" (UID: \"b7340801-9a36-4fe1-be78-44e58bb71d75\") "
May 25 13:27:26.223135 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.222997    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7340801-9a36-4fe1-be78-44e58bb71d75-openshift-service-ca-bundle\") pod \"b7340801-9a36-4fe1-be78-44e58bb71d75\" (UID: \"b7340801-9a36-4fe1-be78-44e58bb71d75\") "
May 25 13:27:26.223642 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.223613    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7340801-9a36-4fe1-be78-44e58bb71d75-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "b7340801-9a36-4fe1-be78-44e58bb71d75" (UID: "b7340801-9a36-4fe1-be78-44e58bb71d75"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:27:26.226073 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.226042    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7340801-9a36-4fe1-be78-44e58bb71d75-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b7340801-9a36-4fe1-be78-44e58bb71d75" (UID: "b7340801-9a36-4fe1-be78-44e58bb71d75"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:27:26.262846 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.262810    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb"]
May 25 13:27:26.263266 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.263250    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7340801-9a36-4fe1-be78-44e58bb71d75" containerName="sequence-graph-9ee57"
May 25 13:27:26.263266 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.263267    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7340801-9a36-4fe1-be78-44e58bb71d75" containerName="sequence-graph-9ee57"
May 25 13:27:26.263398 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.263285    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce196080-f42b-4918-b6f6-f42da3537e67" containerName="kube-rbac-proxy"
May 25 13:27:26.263398 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.263291    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce196080-f42b-4918-b6f6-f42da3537e67" containerName="kube-rbac-proxy"
May 25 13:27:26.263398 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.263306    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce196080-f42b-4918-b6f6-f42da3537e67" containerName="kserve-container"
May 25 13:27:26.263398 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.263312    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce196080-f42b-4918-b6f6-f42da3537e67" containerName="kserve-container"
May 25 13:27:26.263398 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.263361    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="ce196080-f42b-4918-b6f6-f42da3537e67" containerName="kserve-container"
May 25 13:27:26.263398 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.263394    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="ce196080-f42b-4918-b6f6-f42da3537e67" containerName="kube-rbac-proxy"
May 25 13:27:26.263683 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.263403    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="b7340801-9a36-4fe1-be78-44e58bb71d75" containerName="sequence-graph-9ee57"
May 25 13:27:26.266822 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.266800    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb"
May 25 13:27:26.269231 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.269187    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-3e647-serving-cert\""
May 25 13:27:26.269744 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.269722    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-3e647-kube-rbac-proxy-sar-config\""
May 25 13:27:26.277998 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.277970    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb"]
May 25 13:27:26.323497 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.323464    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6c6b443-e7cb-4333-8376-42f13d8e74a0-openshift-service-ca-bundle\") pod \"ensemble-graph-3e647-56d7ff6db-8dvvb\" (UID: \"c6c6b443-e7cb-4333-8376-42f13d8e74a0\") " pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb"
May 25 13:27:26.323668 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.323529    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6c6b443-e7cb-4333-8376-42f13d8e74a0-proxy-tls\") pod \"ensemble-graph-3e647-56d7ff6db-8dvvb\" (UID: \"c6c6b443-e7cb-4333-8376-42f13d8e74a0\") " pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb"
May 25 13:27:26.323668 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.323583    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7340801-9a36-4fe1-be78-44e58bb71d75-openshift-service-ca-bundle\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:27:26.323668 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.323598    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7340801-9a36-4fe1-be78-44e58bb71d75-proxy-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:27:26.423952 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.423909    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6c6b443-e7cb-4333-8376-42f13d8e74a0-openshift-service-ca-bundle\") pod \"ensemble-graph-3e647-56d7ff6db-8dvvb\" (UID: \"c6c6b443-e7cb-4333-8376-42f13d8e74a0\") " pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb"
May 25 13:27:26.424160 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.423970    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6c6b443-e7cb-4333-8376-42f13d8e74a0-proxy-tls\") pod \"ensemble-graph-3e647-56d7ff6db-8dvvb\" (UID: \"c6c6b443-e7cb-4333-8376-42f13d8e74a0\") " pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb"
May 25 13:27:26.424160 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:27:26.424086    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-3e647-serving-cert: secret "ensemble-graph-3e647-serving-cert" not found
May 25 13:27:26.424319 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:27:26.424182    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6c6b443-e7cb-4333-8376-42f13d8e74a0-proxy-tls podName:c6c6b443-e7cb-4333-8376-42f13d8e74a0 nodeName:}" failed. No retries permitted until 2026-05-25 13:27:26.924146082 +0000 UTC m=+1362.926887822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c6c6b443-e7cb-4333-8376-42f13d8e74a0-proxy-tls") pod "ensemble-graph-3e647-56d7ff6db-8dvvb" (UID: "c6c6b443-e7cb-4333-8376-42f13d8e74a0") : secret "ensemble-graph-3e647-serving-cert" not found
May 25 13:27:26.424652 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.424629    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6c6b443-e7cb-4333-8376-42f13d8e74a0-openshift-service-ca-bundle\") pod \"ensemble-graph-3e647-56d7ff6db-8dvvb\" (UID: \"c6c6b443-e7cb-4333-8376-42f13d8e74a0\") " pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb"
May 25 13:27:26.928927 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.928839    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6c6b443-e7cb-4333-8376-42f13d8e74a0-proxy-tls\") pod \"ensemble-graph-3e647-56d7ff6db-8dvvb\" (UID: \"c6c6b443-e7cb-4333-8376-42f13d8e74a0\") " pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb"
May 25 13:27:26.931255 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:26.931219    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6c6b443-e7cb-4333-8376-42f13d8e74a0-proxy-tls\") pod \"ensemble-graph-3e647-56d7ff6db-8dvvb\" (UID: \"c6c6b443-e7cb-4333-8376-42f13d8e74a0\") " pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb"
May 25 13:27:27.180088 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:27.179976    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb"
May 25 13:27:27.187932 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:27.187894    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw" event={"ID":"b7340801-9a36-4fe1-be78-44e58bb71d75","Type":"ContainerDied","Data":"d27667aedbab4d66832bc6048db3bd1a813799d5df7dc01e2718a806928d04df"}
May 25 13:27:27.188081 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:27.187938    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw"
May 25 13:27:27.188081 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:27.187955    2569 scope.go:117] "RemoveContainer" containerID="5d4e47856025e9011e1d3d3e226018cbf7e380bc0b810cf1643205926b7591da"
May 25 13:27:27.214683 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:27.214651    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw"]
May 25 13:27:27.218083 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:27.218055    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9ee57-67b68b489d-bvwfw"]
May 25 13:27:27.313870 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:27.313842    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb"]
May 25 13:27:27.316459 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:27:27.316424    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c6b443_e7cb_4333_8376_42f13d8e74a0.slice/crio-d3ac128ec4e3cf32bdb05e29339689f5da0cae24936dbc85d7227314f06bbe32 WatchSource:0}: Error finding container d3ac128ec4e3cf32bdb05e29339689f5da0cae24936dbc85d7227314f06bbe32: Status 404 returned error can't find the container with id d3ac128ec4e3cf32bdb05e29339689f5da0cae24936dbc85d7227314f06bbe32
May 25 13:27:28.192559 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:28.192524    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb" event={"ID":"c6c6b443-e7cb-4333-8376-42f13d8e74a0","Type":"ContainerStarted","Data":"ff29701e37057508e223e14f3a4094848430a9f8622796e9ccc14d92a45a5cc3"}
May 25 13:27:28.192559 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:28.192559    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb" event={"ID":"c6c6b443-e7cb-4333-8376-42f13d8e74a0","Type":"ContainerStarted","Data":"d3ac128ec4e3cf32bdb05e29339689f5da0cae24936dbc85d7227314f06bbe32"}
May 25 13:27:28.193067 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:28.192590    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb"
May 25 13:27:28.210818 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:28.210769    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb" podStartSLOduration=2.210752764 podStartE2EDuration="2.210752764s" podCreationTimestamp="2026-05-25 13:27:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:27:28.209947761 +0000 UTC m=+1364.212689514" watchObservedRunningTime="2026-05-25 13:27:28.210752764 +0000 UTC m=+1364.213494518"
May 25 13:27:28.597651 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:28.597619    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7340801-9a36-4fe1-be78-44e58bb71d75" path="/var/lib/kubelet/pods/b7340801-9a36-4fe1-be78-44e58bb71d75/volumes"
May 25 13:27:34.094221 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:34.094156    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt" podUID="f074234f-c787-4e18-ab9c-3cf3e375eecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 13:27:34.202794 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:34.202762    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb"
May 25 13:27:44.094463 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:44.094423    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"
May 25 13:27:56.311088 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:56.311044    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb"]
May 25 13:27:56.315479 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:56.315459    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb"
May 25 13:27:56.318930 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:56.318897    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-72c4b-kube-rbac-proxy-sar-config\""
May 25 13:27:56.319071 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:56.319004    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-72c4b-serving-cert\""
May 25 13:27:56.327542 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:56.327520    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb"]
May 25 13:27:56.387240 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:56.387206    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa28c358-d76d-46fd-ae84-1f31f46e7fe7-proxy-tls\") pod \"sequence-graph-72c4b-7c78954875-qkckb\" (UID: \"aa28c358-d76d-46fd-ae84-1f31f46e7fe7\") " pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb"
May 25 13:27:56.387428 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:56.387290    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa28c358-d76d-46fd-ae84-1f31f46e7fe7-openshift-service-ca-bundle\") pod \"sequence-graph-72c4b-7c78954875-qkckb\" (UID: \"aa28c358-d76d-46fd-ae84-1f31f46e7fe7\") " pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb"
May 25 13:27:56.487864 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:56.487818    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa28c358-d76d-46fd-ae84-1f31f46e7fe7-openshift-service-ca-bundle\") pod \"sequence-graph-72c4b-7c78954875-qkckb\" (UID: \"aa28c358-d76d-46fd-ae84-1f31f46e7fe7\") " pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb"
May 25 13:27:56.488046 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:56.487904    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa28c358-d76d-46fd-ae84-1f31f46e7fe7-proxy-tls\") pod \"sequence-graph-72c4b-7c78954875-qkckb\" (UID: \"aa28c358-d76d-46fd-ae84-1f31f46e7fe7\") " pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb"
May 25 13:27:56.488526 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:56.488502    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa28c358-d76d-46fd-ae84-1f31f46e7fe7-openshift-service-ca-bundle\") pod \"sequence-graph-72c4b-7c78954875-qkckb\" (UID: \"aa28c358-d76d-46fd-ae84-1f31f46e7fe7\") " pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb"
May 25 13:27:56.490403 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:56.490370    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa28c358-d76d-46fd-ae84-1f31f46e7fe7-proxy-tls\") pod \"sequence-graph-72c4b-7c78954875-qkckb\" (UID: \"aa28c358-d76d-46fd-ae84-1f31f46e7fe7\") " pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb"
May 25 13:27:56.625306 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:56.625272    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb"
May 25 13:27:56.762352 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:56.762328    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb"]
May 25 13:27:56.765066 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:27:56.765021    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa28c358_d76d_46fd_ae84_1f31f46e7fe7.slice/crio-2ade1dfe797a1c9dfac4c25db3f3eb09e2b475cd0e1299441258f9772ba8cac4 WatchSource:0}: Error finding container 2ade1dfe797a1c9dfac4c25db3f3eb09e2b475cd0e1299441258f9772ba8cac4: Status 404 returned error can't find the container with id 2ade1dfe797a1c9dfac4c25db3f3eb09e2b475cd0e1299441258f9772ba8cac4
May 25 13:27:57.292614 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:57.292576    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb" event={"ID":"aa28c358-d76d-46fd-ae84-1f31f46e7fe7","Type":"ContainerStarted","Data":"4109bdbd3a1ff5d007776e5e874fb88ec1077d3ca4945371b54f024b004ef301"}
May 25 13:27:57.292614 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:57.292614    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb" event={"ID":"aa28c358-d76d-46fd-ae84-1f31f46e7fe7","Type":"ContainerStarted","Data":"2ade1dfe797a1c9dfac4c25db3f3eb09e2b475cd0e1299441258f9772ba8cac4"}
May 25 13:27:57.292853 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:57.292702    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb"
May 25 13:27:57.311922 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:27:57.311866    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb" podStartSLOduration=1.311849203 podStartE2EDuration="1.311849203s" podCreationTimestamp="2026-05-25 13:27:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:27:57.309504239 +0000 UTC m=+1393.312245978" watchObservedRunningTime="2026-05-25 13:27:57.311849203 +0000 UTC m=+1393.314591036"
May 25 13:28:03.301348 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:28:03.301320    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb"
May 25 13:29:45.623585 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:29:45.623553    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/1.log"
May 25 13:29:45.625461 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:29:45.625436    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/1.log"
May 25 13:29:45.628782 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:29:45.628766    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:29:45.630324 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:29:45.630304    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:34:45.652016 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:34:45.651987    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/1.log"
May 25 13:34:45.654648 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:34:45.654615    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/1.log"
May 25 13:34:45.656743 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:34:45.656718    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:34:45.659541 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:34:45.659523    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:35:40.890804 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:40.890773    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb"]
May 25 13:35:40.891374 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:40.891006    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb" podUID="c6c6b443-e7cb-4333-8376-42f13d8e74a0" containerName="ensemble-graph-3e647" containerID="cri-o://ff29701e37057508e223e14f3a4094848430a9f8622796e9ccc14d92a45a5cc3" gracePeriod=30
May 25 13:35:41.098501 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.098467    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"]
May 25 13:35:41.098772 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.098750    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6" podUID="09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3" containerName="kserve-container" containerID="cri-o://4d94ce926ebde1145db97c1ee5741c12a6621bc2b7d7aac65af62750730237ca" gracePeriod=30
May 25 13:35:41.098871 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.098811    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6" podUID="09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3" containerName="kube-rbac-proxy" containerID="cri-o://399246ec781a2a8195ba0f93d4345a619578d27c032ad42604f6778bce8ab3fc" gracePeriod=30
May 25 13:35:41.184571 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.184477    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"]
May 25 13:35:41.188418 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.188390    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"
May 25 13:35:41.191052 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.191025    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-35687-kube-rbac-proxy-sar-config\""
May 25 13:35:41.191221 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.191032    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-35687-predictor-serving-cert\""
May 25 13:35:41.198522 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.198490    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"]
May 25 13:35:41.247783 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.247743    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjk46\" (UniqueName: \"kubernetes.io/projected/b0756412-f126-47cc-b3cf-fffc1e17e5be-kube-api-access-xjk46\") pod \"error-404-isvc-35687-predictor-d8849b54c-bcggv\" (UID: \"b0756412-f126-47cc-b3cf-fffc1e17e5be\") " pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"
May 25 13:35:41.247955 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.247790    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0756412-f126-47cc-b3cf-fffc1e17e5be-proxy-tls\") pod \"error-404-isvc-35687-predictor-d8849b54c-bcggv\" (UID: \"b0756412-f126-47cc-b3cf-fffc1e17e5be\") " pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"
May 25 13:35:41.247955 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.247857    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-35687-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0756412-f126-47cc-b3cf-fffc1e17e5be-error-404-isvc-35687-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-35687-predictor-d8849b54c-bcggv\" (UID: \"b0756412-f126-47cc-b3cf-fffc1e17e5be\") " pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"
May 25 13:35:41.348820 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.348775    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0756412-f126-47cc-b3cf-fffc1e17e5be-proxy-tls\") pod \"error-404-isvc-35687-predictor-d8849b54c-bcggv\" (UID: \"b0756412-f126-47cc-b3cf-fffc1e17e5be\") " pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"
May 25 13:35:41.349000 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.348840    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-35687-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0756412-f126-47cc-b3cf-fffc1e17e5be-error-404-isvc-35687-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-35687-predictor-d8849b54c-bcggv\" (UID: \"b0756412-f126-47cc-b3cf-fffc1e17e5be\") " pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"
May 25 13:35:41.349000 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.348902    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xjk46\" (UniqueName: \"kubernetes.io/projected/b0756412-f126-47cc-b3cf-fffc1e17e5be-kube-api-access-xjk46\") pod \"error-404-isvc-35687-predictor-d8849b54c-bcggv\" (UID: \"b0756412-f126-47cc-b3cf-fffc1e17e5be\") " pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"
May 25 13:35:41.349680 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.349651    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-35687-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0756412-f126-47cc-b3cf-fffc1e17e5be-error-404-isvc-35687-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-35687-predictor-d8849b54c-bcggv\" (UID: \"b0756412-f126-47cc-b3cf-fffc1e17e5be\") " pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"
May 25 13:35:41.351513 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.351487    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0756412-f126-47cc-b3cf-fffc1e17e5be-proxy-tls\") pod \"error-404-isvc-35687-predictor-d8849b54c-bcggv\" (UID: \"b0756412-f126-47cc-b3cf-fffc1e17e5be\") " pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"
May 25 13:35:41.358776 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.358751    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjk46\" (UniqueName: \"kubernetes.io/projected/b0756412-f126-47cc-b3cf-fffc1e17e5be-kube-api-access-xjk46\") pod \"error-404-isvc-35687-predictor-d8849b54c-bcggv\" (UID: \"b0756412-f126-47cc-b3cf-fffc1e17e5be\") " pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"
May 25 13:35:41.501527 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.501428    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"
May 25 13:35:41.637498 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.637475    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"]
May 25 13:35:41.640109 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:35:41.640086    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0756412_f126_47cc_b3cf_fffc1e17e5be.slice/crio-b00652429cd06f707fc3a736526fddc5ab3c31929f91cf5cdf16859b4e67bdc1 WatchSource:0}: Error finding container b00652429cd06f707fc3a736526fddc5ab3c31929f91cf5cdf16859b4e67bdc1: Status 404 returned error can't find the container with id b00652429cd06f707fc3a736526fddc5ab3c31929f91cf5cdf16859b4e67bdc1
May 25 13:35:41.641822 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.641801    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:35:41.892846 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.892810    2569 generic.go:358] "Generic (PLEG): container finished" podID="09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3" containerID="399246ec781a2a8195ba0f93d4345a619578d27c032ad42604f6778bce8ab3fc" exitCode=2
May 25 13:35:41.893254 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.892879    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6" event={"ID":"09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3","Type":"ContainerDied","Data":"399246ec781a2a8195ba0f93d4345a619578d27c032ad42604f6778bce8ab3fc"}
May 25 13:35:41.894465 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.894443    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv" event={"ID":"b0756412-f126-47cc-b3cf-fffc1e17e5be","Type":"ContainerStarted","Data":"63f35aff54159a1d9ceaf3057162f3c5b77ae1cdd72fe1db18ec65b0cae67179"}
May 25 13:35:41.894571 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.894473    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv" event={"ID":"b0756412-f126-47cc-b3cf-fffc1e17e5be","Type":"ContainerStarted","Data":"a44d95dad5cfc87ae1ebb7b8a4fe9d6f89ba1f0e99167e50c97beb749b9d3360"}
May 25 13:35:41.894571 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.894488    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv" event={"ID":"b0756412-f126-47cc-b3cf-fffc1e17e5be","Type":"ContainerStarted","Data":"b00652429cd06f707fc3a736526fddc5ab3c31929f91cf5cdf16859b4e67bdc1"}
May 25 13:35:41.894655 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.894613    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"
May 25 13:35:41.914072 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:41.914023    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv" podStartSLOduration=0.914009208 podStartE2EDuration="914.009208ms" podCreationTimestamp="2026-05-25 13:35:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:35:41.912158571 +0000 UTC m=+1857.914900326" watchObservedRunningTime="2026-05-25 13:35:41.914009208 +0000 UTC m=+1857.916750970"
May 25 13:35:42.898026 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:42.897998    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"
May 25 13:35:42.899338 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:42.899297    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv" podUID="b0756412-f126-47cc-b3cf-fffc1e17e5be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 25 13:35:42.962574 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:42.962531    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6" podUID="09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
May 25 13:35:42.967886 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:42.967855    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6" podUID="09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 25 13:35:43.901864 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:43.901818    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv" podUID="b0756412-f126-47cc-b3cf-fffc1e17e5be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 25 13:35:44.200517 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:44.200414    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb" podUID="c6c6b443-e7cb-4333-8376-42f13d8e74a0" containerName="ensemble-graph-3e647" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:35:44.350384 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:44.350358    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"
May 25 13:35:44.479380 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:44.479298    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3-proxy-tls\") pod \"09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3\" (UID: \"09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3\") "
May 25 13:35:44.479380 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:44.479375    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzmxg\" (UniqueName: \"kubernetes.io/projected/09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3-kube-api-access-pzmxg\") pod \"09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3\" (UID: \"09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3\") "
May 25 13:35:44.479583 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:44.479478    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-3e647-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3-error-404-isvc-3e647-kube-rbac-proxy-sar-config\") pod \"09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3\" (UID: \"09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3\") "
May 25 13:35:44.479862 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:44.479836    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3-error-404-isvc-3e647-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-3e647-kube-rbac-proxy-sar-config") pod "09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3" (UID: "09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3"). InnerVolumeSpecName "error-404-isvc-3e647-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:35:44.481380 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:44.481360    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3" (UID: "09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:35:44.481450 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:44.481420    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3-kube-api-access-pzmxg" (OuterVolumeSpecName: "kube-api-access-pzmxg") pod "09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3" (UID: "09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3"). InnerVolumeSpecName "kube-api-access-pzmxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:35:44.580496 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:44.580459    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3-proxy-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:35:44.580496 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:44.580492    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzmxg\" (UniqueName: \"kubernetes.io/projected/09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3-kube-api-access-pzmxg\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:35:44.580496 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:44.580504    2569 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-3e647-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3-error-404-isvc-3e647-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:35:44.697956 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:44.697925    2569 scope.go:117] "RemoveContainer" containerID="399246ec781a2a8195ba0f93d4345a619578d27c032ad42604f6778bce8ab3fc"
May 25 13:35:44.706285 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:44.706257    2569 scope.go:117] "RemoveContainer" containerID="4d94ce926ebde1145db97c1ee5741c12a6621bc2b7d7aac65af62750730237ca"
May 25 13:35:44.905091 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:44.905060    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"
May 25 13:35:44.905527 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:44.905057    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6" event={"ID":"09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3","Type":"ContainerDied","Data":"4d94ce926ebde1145db97c1ee5741c12a6621bc2b7d7aac65af62750730237ca"}
May 25 13:35:44.905527 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:44.905196    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6" event={"ID":"09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3","Type":"ContainerDied","Data":"b65857942350b0d588127504cc7b9ee46f5377f0b5d057277c01c9b066ce360c"}
May 25 13:35:44.935914 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:44.935878    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"]
May 25 13:35:44.939936 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:44.939906    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3e647-predictor-9899bf78c-4dxp6"]
May 25 13:35:46.597573 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:46.597537    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3" path="/var/lib/kubelet/pods/09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3/volumes"
May 25 13:35:48.906100 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:48.906070    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"
May 25 13:35:48.906592 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:48.906539    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv" podUID="b0756412-f126-47cc-b3cf-fffc1e17e5be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 25 13:35:49.200514 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:49.200426    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb" podUID="c6c6b443-e7cb-4333-8376-42f13d8e74a0" containerName="ensemble-graph-3e647" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:35:54.200403 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:54.200358    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb" podUID="c6c6b443-e7cb-4333-8376-42f13d8e74a0" containerName="ensemble-graph-3e647" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:35:54.200874 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:54.200502    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb"
May 25 13:35:58.907109 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:58.907069    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv" podUID="b0756412-f126-47cc-b3cf-fffc1e17e5be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 25 13:35:59.200257 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:35:59.200146    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb" podUID="c6c6b443-e7cb-4333-8376-42f13d8e74a0" containerName="ensemble-graph-3e647" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:36:04.200370 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:04.200278    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb" podUID="c6c6b443-e7cb-4333-8376-42f13d8e74a0" containerName="ensemble-graph-3e647" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:36:08.906845 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:08.906800    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv" podUID="b0756412-f126-47cc-b3cf-fffc1e17e5be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 25 13:36:09.200608 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:09.200517    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb" podUID="c6c6b443-e7cb-4333-8376-42f13d8e74a0" containerName="ensemble-graph-3e647" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:36:10.941639 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:10.941600    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb"]
May 25 13:36:10.942109 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:10.941934    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb" podUID="aa28c358-d76d-46fd-ae84-1f31f46e7fe7" containerName="sequence-graph-72c4b" containerID="cri-o://4109bdbd3a1ff5d007776e5e874fb88ec1077d3ca4945371b54f024b004ef301" gracePeriod=30
May 25 13:36:11.004385 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.004348    2569 generic.go:358] "Generic (PLEG): container finished" podID="c6c6b443-e7cb-4333-8376-42f13d8e74a0" containerID="ff29701e37057508e223e14f3a4094848430a9f8622796e9ccc14d92a45a5cc3" exitCode=0
May 25 13:36:11.004562 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.004425    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb" event={"ID":"c6c6b443-e7cb-4333-8376-42f13d8e74a0","Type":"ContainerDied","Data":"ff29701e37057508e223e14f3a4094848430a9f8622796e9ccc14d92a45a5cc3"}
May 25 13:36:11.058196 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.058153    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb"
May 25 13:36:11.095880 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.095842    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6c6b443-e7cb-4333-8376-42f13d8e74a0-proxy-tls\") pod \"c6c6b443-e7cb-4333-8376-42f13d8e74a0\" (UID: \"c6c6b443-e7cb-4333-8376-42f13d8e74a0\") "
May 25 13:36:11.097972 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.097940    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c6b443-e7cb-4333-8376-42f13d8e74a0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c6c6b443-e7cb-4333-8376-42f13d8e74a0" (UID: "c6c6b443-e7cb-4333-8376-42f13d8e74a0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:36:11.165081 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.164952    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"]
May 25 13:36:11.165292 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.165256    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt" podUID="f074234f-c787-4e18-ab9c-3cf3e375eecb" containerName="kserve-container" containerID="cri-o://0cbcb38cb3a58f051b3f6d61261a3089f865ed89323764a6fe98f27c7872e3d4" gracePeriod=30
May 25 13:36:11.165378 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.165321    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt" podUID="f074234f-c787-4e18-ab9c-3cf3e375eecb" containerName="kube-rbac-proxy" containerID="cri-o://afda7a17a6650bbf9b4c5616f8b8ad20eae73a2cd8e4d1fb58e997b1a9624e1d" gracePeriod=30
May 25 13:36:11.196781 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.196734    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6c6b443-e7cb-4333-8376-42f13d8e74a0-openshift-service-ca-bundle\") pod \"c6c6b443-e7cb-4333-8376-42f13d8e74a0\" (UID: \"c6c6b443-e7cb-4333-8376-42f13d8e74a0\") "
May 25 13:36:11.196959 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.196917    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6c6b443-e7cb-4333-8376-42f13d8e74a0-proxy-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:36:11.197104 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.197079    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c6b443-e7cb-4333-8376-42f13d8e74a0-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c6c6b443-e7cb-4333-8376-42f13d8e74a0" (UID: "c6c6b443-e7cb-4333-8376-42f13d8e74a0"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:36:11.214552 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.214516    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"]
May 25 13:36:11.214907 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.214894    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c6b443-e7cb-4333-8376-42f13d8e74a0" containerName="ensemble-graph-3e647"
May 25 13:36:11.214907 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.214908    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c6b443-e7cb-4333-8376-42f13d8e74a0" containerName="ensemble-graph-3e647"
May 25 13:36:11.214992 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.214925    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3" containerName="kserve-container"
May 25 13:36:11.214992 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.214931    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3" containerName="kserve-container"
May 25 13:36:11.214992 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.214938    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3" containerName="kube-rbac-proxy"
May 25 13:36:11.214992 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.214944    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3" containerName="kube-rbac-proxy"
May 25 13:36:11.215205 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.214999    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3" containerName="kserve-container"
May 25 13:36:11.215205 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.215008    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="c6c6b443-e7cb-4333-8376-42f13d8e74a0" containerName="ensemble-graph-3e647"
May 25 13:36:11.215205 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.215016    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="09e86a6b-5c8f-4e71-b465-cbe3c22c9ca3" containerName="kube-rbac-proxy"
May 25 13:36:11.218614 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.218597    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"
May 25 13:36:11.222928 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.222901    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-12313-predictor-serving-cert\""
May 25 13:36:11.223072 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.223019    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-12313-kube-rbac-proxy-sar-config\""
May 25 13:36:11.232585 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.232552    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"]
May 25 13:36:11.298189 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.298132    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-12313-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7e53380-9d1c-42f5-bbc3-4266185be4a0-error-404-isvc-12313-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-12313-predictor-7746bc869-vlbsc\" (UID: \"a7e53380-9d1c-42f5-bbc3-4266185be4a0\") " pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"
May 25 13:36:11.298189 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.298193    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7e53380-9d1c-42f5-bbc3-4266185be4a0-proxy-tls\") pod \"error-404-isvc-12313-predictor-7746bc869-vlbsc\" (UID: \"a7e53380-9d1c-42f5-bbc3-4266185be4a0\") " pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"
May 25 13:36:11.298381 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.298303    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg5g5\" (UniqueName: \"kubernetes.io/projected/a7e53380-9d1c-42f5-bbc3-4266185be4a0-kube-api-access-hg5g5\") pod \"error-404-isvc-12313-predictor-7746bc869-vlbsc\" (UID: \"a7e53380-9d1c-42f5-bbc3-4266185be4a0\") " pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"
May 25 13:36:11.298449 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.298424    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6c6b443-e7cb-4333-8376-42f13d8e74a0-openshift-service-ca-bundle\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:36:11.398971 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.398933    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-12313-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7e53380-9d1c-42f5-bbc3-4266185be4a0-error-404-isvc-12313-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-12313-predictor-7746bc869-vlbsc\" (UID: \"a7e53380-9d1c-42f5-bbc3-4266185be4a0\") " pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"
May 25 13:36:11.399238 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.398990    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7e53380-9d1c-42f5-bbc3-4266185be4a0-proxy-tls\") pod \"error-404-isvc-12313-predictor-7746bc869-vlbsc\" (UID: \"a7e53380-9d1c-42f5-bbc3-4266185be4a0\") " pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"
May 25 13:36:11.399238 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.399068    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hg5g5\" (UniqueName: \"kubernetes.io/projected/a7e53380-9d1c-42f5-bbc3-4266185be4a0-kube-api-access-hg5g5\") pod \"error-404-isvc-12313-predictor-7746bc869-vlbsc\" (UID: \"a7e53380-9d1c-42f5-bbc3-4266185be4a0\") " pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"
May 25 13:36:11.399238 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:36:11.399153    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-12313-predictor-serving-cert: secret "error-404-isvc-12313-predictor-serving-cert" not found
May 25 13:36:11.399443 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:36:11.399255    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e53380-9d1c-42f5-bbc3-4266185be4a0-proxy-tls podName:a7e53380-9d1c-42f5-bbc3-4266185be4a0 nodeName:}" failed. No retries permitted until 2026-05-25 13:36:11.899233418 +0000 UTC m=+1887.901975168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a7e53380-9d1c-42f5-bbc3-4266185be4a0-proxy-tls") pod "error-404-isvc-12313-predictor-7746bc869-vlbsc" (UID: "a7e53380-9d1c-42f5-bbc3-4266185be4a0") : secret "error-404-isvc-12313-predictor-serving-cert" not found
May 25 13:36:11.399652 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.399632    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-12313-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7e53380-9d1c-42f5-bbc3-4266185be4a0-error-404-isvc-12313-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-12313-predictor-7746bc869-vlbsc\" (UID: \"a7e53380-9d1c-42f5-bbc3-4266185be4a0\") " pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"
May 25 13:36:11.413914 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.413889    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg5g5\" (UniqueName: \"kubernetes.io/projected/a7e53380-9d1c-42f5-bbc3-4266185be4a0-kube-api-access-hg5g5\") pod \"error-404-isvc-12313-predictor-7746bc869-vlbsc\" (UID: \"a7e53380-9d1c-42f5-bbc3-4266185be4a0\") " pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"
May 25 13:36:11.903972 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.903932    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7e53380-9d1c-42f5-bbc3-4266185be4a0-proxy-tls\") pod \"error-404-isvc-12313-predictor-7746bc869-vlbsc\" (UID: \"a7e53380-9d1c-42f5-bbc3-4266185be4a0\") " pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"
May 25 13:36:11.906552 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:11.906529    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7e53380-9d1c-42f5-bbc3-4266185be4a0-proxy-tls\") pod \"error-404-isvc-12313-predictor-7746bc869-vlbsc\" (UID: \"a7e53380-9d1c-42f5-bbc3-4266185be4a0\") " pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"
May 25 13:36:12.009549 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:12.009512    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb"
May 25 13:36:12.010031 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:12.009545    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb" event={"ID":"c6c6b443-e7cb-4333-8376-42f13d8e74a0","Type":"ContainerDied","Data":"d3ac128ec4e3cf32bdb05e29339689f5da0cae24936dbc85d7227314f06bbe32"}
May 25 13:36:12.010031 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:12.009601    2569 scope.go:117] "RemoveContainer" containerID="ff29701e37057508e223e14f3a4094848430a9f8622796e9ccc14d92a45a5cc3"
May 25 13:36:12.011514 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:12.011487    2569 generic.go:358] "Generic (PLEG): container finished" podID="f074234f-c787-4e18-ab9c-3cf3e375eecb" containerID="afda7a17a6650bbf9b4c5616f8b8ad20eae73a2cd8e4d1fb58e997b1a9624e1d" exitCode=2
May 25 13:36:12.011617 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:12.011535    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt" event={"ID":"f074234f-c787-4e18-ab9c-3cf3e375eecb","Type":"ContainerDied","Data":"afda7a17a6650bbf9b4c5616f8b8ad20eae73a2cd8e4d1fb58e997b1a9624e1d"}
May 25 13:36:12.035406 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:12.035373    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb"]
May 25 13:36:12.041611 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:12.041583    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-3e647-56d7ff6db-8dvvb"]
May 25 13:36:12.129700 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:12.129650    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"
May 25 13:36:12.268977 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:12.268934    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"]
May 25 13:36:12.271722 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:36:12.271694    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e53380_9d1c_42f5_bbc3_4266185be4a0.slice/crio-6d6c7db41ff6f2194f1daf45e3aa94ad7b949a8b10716581d0e14438b10353ce WatchSource:0}: Error finding container 6d6c7db41ff6f2194f1daf45e3aa94ad7b949a8b10716581d0e14438b10353ce: Status 404 returned error can't find the container with id 6d6c7db41ff6f2194f1daf45e3aa94ad7b949a8b10716581d0e14438b10353ce
May 25 13:36:12.597868 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:12.597826    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6c6b443-e7cb-4333-8376-42f13d8e74a0" path="/var/lib/kubelet/pods/c6c6b443-e7cb-4333-8376-42f13d8e74a0/volumes"
May 25 13:36:13.016544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:13.016442    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc" event={"ID":"a7e53380-9d1c-42f5-bbc3-4266185be4a0","Type":"ContainerStarted","Data":"6e9d5cc18fbe7d13369784000fe9f8d97c3be253e48424a9990702a3204b1c9b"}
May 25 13:36:13.016544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:13.016484    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc" event={"ID":"a7e53380-9d1c-42f5-bbc3-4266185be4a0","Type":"ContainerStarted","Data":"65eed7ee6ce35f4bd51ba461b1bc93f48cefc31b8a179ec35dd835d249b8d35c"}
May 25 13:36:13.016544 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:13.016499    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc" event={"ID":"a7e53380-9d1c-42f5-bbc3-4266185be4a0","Type":"ContainerStarted","Data":"6d6c7db41ff6f2194f1daf45e3aa94ad7b949a8b10716581d0e14438b10353ce"}
May 25 13:36:13.017100 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:13.016587    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"
May 25 13:36:13.017100 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:13.016618    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"
May 25 13:36:13.018093 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:13.018066    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc" podUID="a7e53380-9d1c-42f5-bbc3-4266185be4a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
May 25 13:36:13.034579 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:13.034529    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc" podStartSLOduration=2.034515207 podStartE2EDuration="2.034515207s" podCreationTimestamp="2026-05-25 13:36:11 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:36:13.033356568 +0000 UTC m=+1889.036098333" watchObservedRunningTime="2026-05-25 13:36:13.034515207 +0000 UTC m=+1889.037256960"
May 25 13:36:13.300264 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:13.300151    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb" podUID="aa28c358-d76d-46fd-ae84-1f31f46e7fe7" containerName="sequence-graph-72c4b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:36:14.020503 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:14.020463    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc" podUID="a7e53380-9d1c-42f5-bbc3-4266185be4a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
May 25 13:36:14.089338 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:14.089293    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt" podUID="f074234f-c787-4e18-ab9c-3cf3e375eecb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
May 25 13:36:14.094608 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:14.094577    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt" podUID="f074234f-c787-4e18-ab9c-3cf3e375eecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 13:36:14.615577 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:14.615552    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"
May 25 13:36:14.621045 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:14.621021    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-72c4b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f074234f-c787-4e18-ab9c-3cf3e375eecb-error-404-isvc-72c4b-kube-rbac-proxy-sar-config\") pod \"f074234f-c787-4e18-ab9c-3cf3e375eecb\" (UID: \"f074234f-c787-4e18-ab9c-3cf3e375eecb\") "
May 25 13:36:14.621115 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:14.621058    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f074234f-c787-4e18-ab9c-3cf3e375eecb-proxy-tls\") pod \"f074234f-c787-4e18-ab9c-3cf3e375eecb\" (UID: \"f074234f-c787-4e18-ab9c-3cf3e375eecb\") "
May 25 13:36:14.621115 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:14.621098    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84nnw\" (UniqueName: \"kubernetes.io/projected/f074234f-c787-4e18-ab9c-3cf3e375eecb-kube-api-access-84nnw\") pod \"f074234f-c787-4e18-ab9c-3cf3e375eecb\" (UID: \"f074234f-c787-4e18-ab9c-3cf3e375eecb\") "
May 25 13:36:14.621420 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:14.621379    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f074234f-c787-4e18-ab9c-3cf3e375eecb-error-404-isvc-72c4b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-72c4b-kube-rbac-proxy-sar-config") pod "f074234f-c787-4e18-ab9c-3cf3e375eecb" (UID: "f074234f-c787-4e18-ab9c-3cf3e375eecb"). InnerVolumeSpecName "error-404-isvc-72c4b-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:36:14.623231 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:14.623196    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f074234f-c787-4e18-ab9c-3cf3e375eecb-kube-api-access-84nnw" (OuterVolumeSpecName: "kube-api-access-84nnw") pod "f074234f-c787-4e18-ab9c-3cf3e375eecb" (UID: "f074234f-c787-4e18-ab9c-3cf3e375eecb"). InnerVolumeSpecName "kube-api-access-84nnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:36:14.623335 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:14.623227    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f074234f-c787-4e18-ab9c-3cf3e375eecb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f074234f-c787-4e18-ab9c-3cf3e375eecb" (UID: "f074234f-c787-4e18-ab9c-3cf3e375eecb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:36:14.721734 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:14.721642    2569 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-72c4b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f074234f-c787-4e18-ab9c-3cf3e375eecb-error-404-isvc-72c4b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:36:14.721734 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:14.721675    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f074234f-c787-4e18-ab9c-3cf3e375eecb-proxy-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:36:14.721734 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:14.721688    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84nnw\" (UniqueName: \"kubernetes.io/projected/f074234f-c787-4e18-ab9c-3cf3e375eecb-kube-api-access-84nnw\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:36:15.025757 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:15.025669    2569 generic.go:358] "Generic (PLEG): container finished" podID="f074234f-c787-4e18-ab9c-3cf3e375eecb" containerID="0cbcb38cb3a58f051b3f6d61261a3089f865ed89323764a6fe98f27c7872e3d4" exitCode=0
May 25 13:36:15.025757 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:15.025727    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt" event={"ID":"f074234f-c787-4e18-ab9c-3cf3e375eecb","Type":"ContainerDied","Data":"0cbcb38cb3a58f051b3f6d61261a3089f865ed89323764a6fe98f27c7872e3d4"}
May 25 13:36:15.025757 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:15.025756    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt" event={"ID":"f074234f-c787-4e18-ab9c-3cf3e375eecb","Type":"ContainerDied","Data":"3b59742a2e6977b80607396fc45a740c3b91886eec0ba4046db5189722f37cd7"}
May 25 13:36:15.026321 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:15.025773    2569 scope.go:117] "RemoveContainer" containerID="afda7a17a6650bbf9b4c5616f8b8ad20eae73a2cd8e4d1fb58e997b1a9624e1d"
May 25 13:36:15.026321 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:15.025770    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"
May 25 13:36:15.035197 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:15.035158    2569 scope.go:117] "RemoveContainer" containerID="0cbcb38cb3a58f051b3f6d61261a3089f865ed89323764a6fe98f27c7872e3d4"
May 25 13:36:15.042327 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:15.042306    2569 scope.go:117] "RemoveContainer" containerID="afda7a17a6650bbf9b4c5616f8b8ad20eae73a2cd8e4d1fb58e997b1a9624e1d"
May 25 13:36:15.042577 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:36:15.042556    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afda7a17a6650bbf9b4c5616f8b8ad20eae73a2cd8e4d1fb58e997b1a9624e1d\": container with ID starting with afda7a17a6650bbf9b4c5616f8b8ad20eae73a2cd8e4d1fb58e997b1a9624e1d not found: ID does not exist" containerID="afda7a17a6650bbf9b4c5616f8b8ad20eae73a2cd8e4d1fb58e997b1a9624e1d"
May 25 13:36:15.042650 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:15.042590    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afda7a17a6650bbf9b4c5616f8b8ad20eae73a2cd8e4d1fb58e997b1a9624e1d"} err="failed to get container status \"afda7a17a6650bbf9b4c5616f8b8ad20eae73a2cd8e4d1fb58e997b1a9624e1d\": rpc error: code = NotFound desc = could not find container \"afda7a17a6650bbf9b4c5616f8b8ad20eae73a2cd8e4d1fb58e997b1a9624e1d\": container with ID starting with afda7a17a6650bbf9b4c5616f8b8ad20eae73a2cd8e4d1fb58e997b1a9624e1d not found: ID does not exist"
May 25 13:36:15.042650 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:15.042617    2569 scope.go:117] "RemoveContainer" containerID="0cbcb38cb3a58f051b3f6d61261a3089f865ed89323764a6fe98f27c7872e3d4"
May 25 13:36:15.042860 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:36:15.042843    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cbcb38cb3a58f051b3f6d61261a3089f865ed89323764a6fe98f27c7872e3d4\": container with ID starting with 0cbcb38cb3a58f051b3f6d61261a3089f865ed89323764a6fe98f27c7872e3d4 not found: ID does not exist" containerID="0cbcb38cb3a58f051b3f6d61261a3089f865ed89323764a6fe98f27c7872e3d4"
May 25 13:36:15.042900 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:15.042869    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cbcb38cb3a58f051b3f6d61261a3089f865ed89323764a6fe98f27c7872e3d4"} err="failed to get container status \"0cbcb38cb3a58f051b3f6d61261a3089f865ed89323764a6fe98f27c7872e3d4\": rpc error: code = NotFound desc = could not find container \"0cbcb38cb3a58f051b3f6d61261a3089f865ed89323764a6fe98f27c7872e3d4\": container with ID starting with 0cbcb38cb3a58f051b3f6d61261a3089f865ed89323764a6fe98f27c7872e3d4 not found: ID does not exist"
May 25 13:36:15.046753 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:15.046732    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"]
May 25 13:36:15.050916 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:15.050896    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-72c4b-predictor-549bf86d85-lbvpt"]
May 25 13:36:16.597262 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:16.597229    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f074234f-c787-4e18-ab9c-3cf3e375eecb" path="/var/lib/kubelet/pods/f074234f-c787-4e18-ab9c-3cf3e375eecb/volumes"
May 25 13:36:18.300388 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:18.300350    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb" podUID="aa28c358-d76d-46fd-ae84-1f31f46e7fe7" containerName="sequence-graph-72c4b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:36:18.907280 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:18.907235    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv" podUID="b0756412-f126-47cc-b3cf-fffc1e17e5be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 25 13:36:19.026111 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:19.026071    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"
May 25 13:36:19.026708 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:19.026677    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc" podUID="a7e53380-9d1c-42f5-bbc3-4266185be4a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
May 25 13:36:23.299980 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:23.299935    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb" podUID="aa28c358-d76d-46fd-ae84-1f31f46e7fe7" containerName="sequence-graph-72c4b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:36:23.300364 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:23.300044    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb"
May 25 13:36:28.299536 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:28.299495    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb" podUID="aa28c358-d76d-46fd-ae84-1f31f46e7fe7" containerName="sequence-graph-72c4b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:36:28.907837 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:28.907809    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"
May 25 13:36:29.027000 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:29.026955    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc" podUID="a7e53380-9d1c-42f5-bbc3-4266185be4a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
May 25 13:36:33.300196 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:33.300117    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb" podUID="aa28c358-d76d-46fd-ae84-1f31f46e7fe7" containerName="sequence-graph-72c4b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:36:38.299675 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:38.299630    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb" podUID="aa28c358-d76d-46fd-ae84-1f31f46e7fe7" containerName="sequence-graph-72c4b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:36:39.027529 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:39.027487    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc" podUID="a7e53380-9d1c-42f5-bbc3-4266185be4a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
May 25 13:36:41.083313 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.083290    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb"
May 25 13:36:41.114503 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.114466    2569 generic.go:358] "Generic (PLEG): container finished" podID="aa28c358-d76d-46fd-ae84-1f31f46e7fe7" containerID="4109bdbd3a1ff5d007776e5e874fb88ec1077d3ca4945371b54f024b004ef301" exitCode=0
May 25 13:36:41.114681 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.114542    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb"
May 25 13:36:41.114681 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.114552    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb" event={"ID":"aa28c358-d76d-46fd-ae84-1f31f46e7fe7","Type":"ContainerDied","Data":"4109bdbd3a1ff5d007776e5e874fb88ec1077d3ca4945371b54f024b004ef301"}
May 25 13:36:41.114681 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.114592    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb" event={"ID":"aa28c358-d76d-46fd-ae84-1f31f46e7fe7","Type":"ContainerDied","Data":"2ade1dfe797a1c9dfac4c25db3f3eb09e2b475cd0e1299441258f9772ba8cac4"}
May 25 13:36:41.114681 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.114610    2569 scope.go:117] "RemoveContainer" containerID="4109bdbd3a1ff5d007776e5e874fb88ec1077d3ca4945371b54f024b004ef301"
May 25 13:36:41.122841 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.122811    2569 scope.go:117] "RemoveContainer" containerID="4109bdbd3a1ff5d007776e5e874fb88ec1077d3ca4945371b54f024b004ef301"
May 25 13:36:41.123136 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:36:41.123109    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4109bdbd3a1ff5d007776e5e874fb88ec1077d3ca4945371b54f024b004ef301\": container with ID starting with 4109bdbd3a1ff5d007776e5e874fb88ec1077d3ca4945371b54f024b004ef301 not found: ID does not exist" containerID="4109bdbd3a1ff5d007776e5e874fb88ec1077d3ca4945371b54f024b004ef301"
May 25 13:36:41.123253 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.123149    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4109bdbd3a1ff5d007776e5e874fb88ec1077d3ca4945371b54f024b004ef301"} err="failed to get container status \"4109bdbd3a1ff5d007776e5e874fb88ec1077d3ca4945371b54f024b004ef301\": rpc error: code = NotFound desc = could not find container \"4109bdbd3a1ff5d007776e5e874fb88ec1077d3ca4945371b54f024b004ef301\": container with ID starting with 4109bdbd3a1ff5d007776e5e874fb88ec1077d3ca4945371b54f024b004ef301 not found: ID does not exist"
May 25 13:36:41.136853 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.136825    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88"]
May 25 13:36:41.137214 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.137201    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f074234f-c787-4e18-ab9c-3cf3e375eecb" containerName="kserve-container"
May 25 13:36:41.137263 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.137216    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f074234f-c787-4e18-ab9c-3cf3e375eecb" containerName="kserve-container"
May 25 13:36:41.137263 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.137225    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa28c358-d76d-46fd-ae84-1f31f46e7fe7" containerName="sequence-graph-72c4b"
May 25 13:36:41.137263 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.137231    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa28c358-d76d-46fd-ae84-1f31f46e7fe7" containerName="sequence-graph-72c4b"
May 25 13:36:41.137263 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.137249    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f074234f-c787-4e18-ab9c-3cf3e375eecb" containerName="kube-rbac-proxy"
May 25 13:36:41.137263 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.137255    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f074234f-c787-4e18-ab9c-3cf3e375eecb" containerName="kube-rbac-proxy"
May 25 13:36:41.137461 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.137309    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="f074234f-c787-4e18-ab9c-3cf3e375eecb" containerName="kube-rbac-proxy"
May 25 13:36:41.137461 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.137319    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="aa28c358-d76d-46fd-ae84-1f31f46e7fe7" containerName="sequence-graph-72c4b"
May 25 13:36:41.137461 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.137325    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="f074234f-c787-4e18-ab9c-3cf3e375eecb" containerName="kserve-container"
May 25 13:36:41.141275 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.141253    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa28c358-d76d-46fd-ae84-1f31f46e7fe7-openshift-service-ca-bundle\") pod \"aa28c358-d76d-46fd-ae84-1f31f46e7fe7\" (UID: \"aa28c358-d76d-46fd-ae84-1f31f46e7fe7\") "
May 25 13:36:41.141402 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.141295    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa28c358-d76d-46fd-ae84-1f31f46e7fe7-proxy-tls\") pod \"aa28c358-d76d-46fd-ae84-1f31f46e7fe7\" (UID: \"aa28c358-d76d-46fd-ae84-1f31f46e7fe7\") "
May 25 13:36:41.141652 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.141629    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa28c358-d76d-46fd-ae84-1f31f46e7fe7-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "aa28c358-d76d-46fd-ae84-1f31f46e7fe7" (UID: "aa28c358-d76d-46fd-ae84-1f31f46e7fe7"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:36:41.141739 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.141722    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88"
May 25 13:36:41.143352 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.143329    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa28c358-d76d-46fd-ae84-1f31f46e7fe7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aa28c358-d76d-46fd-ae84-1f31f46e7fe7" (UID: "aa28c358-d76d-46fd-ae84-1f31f46e7fe7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:36:41.143948 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.143933    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-35687-serving-cert\""
May 25 13:36:41.143994 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.143933    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-35687-kube-rbac-proxy-sar-config\""
May 25 13:36:41.147743 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.147715    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88"]
May 25 13:36:41.242163 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.242071    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40d16003-4946-4138-bf3a-320fe9605fe6-openshift-service-ca-bundle\") pod \"splitter-graph-35687-79cdffcd67-wpm88\" (UID: \"40d16003-4946-4138-bf3a-320fe9605fe6\") " pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88"
May 25 13:36:41.242163 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.242119    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40d16003-4946-4138-bf3a-320fe9605fe6-proxy-tls\") pod \"splitter-graph-35687-79cdffcd67-wpm88\" (UID: \"40d16003-4946-4138-bf3a-320fe9605fe6\") " pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88"
May 25 13:36:41.242384 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.242275    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa28c358-d76d-46fd-ae84-1f31f46e7fe7-openshift-service-ca-bundle\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:36:41.242384 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.242296    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa28c358-d76d-46fd-ae84-1f31f46e7fe7-proxy-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:36:41.343499 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.343458    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40d16003-4946-4138-bf3a-320fe9605fe6-openshift-service-ca-bundle\") pod \"splitter-graph-35687-79cdffcd67-wpm88\" (UID: \"40d16003-4946-4138-bf3a-320fe9605fe6\") " pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88"
May 25 13:36:41.343681 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.343516    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40d16003-4946-4138-bf3a-320fe9605fe6-proxy-tls\") pod \"splitter-graph-35687-79cdffcd67-wpm88\" (UID: \"40d16003-4946-4138-bf3a-320fe9605fe6\") " pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88"
May 25 13:36:41.344108 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.344075    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40d16003-4946-4138-bf3a-320fe9605fe6-openshift-service-ca-bundle\") pod \"splitter-graph-35687-79cdffcd67-wpm88\" (UID: \"40d16003-4946-4138-bf3a-320fe9605fe6\") " pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88"
May 25 13:36:41.345837 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.345817    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40d16003-4946-4138-bf3a-320fe9605fe6-proxy-tls\") pod \"splitter-graph-35687-79cdffcd67-wpm88\" (UID: \"40d16003-4946-4138-bf3a-320fe9605fe6\") " pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88"
May 25 13:36:41.436307 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.436277    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb"]
May 25 13:36:41.440125 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.440100    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-72c4b-7c78954875-qkckb"]
May 25 13:36:41.459469 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.459447    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88"
May 25 13:36:41.590568 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:41.590535    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88"]
May 25 13:36:41.593785 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:36:41.593754    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d16003_4946_4138_bf3a_320fe9605fe6.slice/crio-fbf0ef7fe2d4c4f5d872c6bd5100f9ffb7df356d14df990e3eedcf80eb647195 WatchSource:0}: Error finding container fbf0ef7fe2d4c4f5d872c6bd5100f9ffb7df356d14df990e3eedcf80eb647195: Status 404 returned error can't find the container with id fbf0ef7fe2d4c4f5d872c6bd5100f9ffb7df356d14df990e3eedcf80eb647195
May 25 13:36:42.119836 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:42.119796    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88" event={"ID":"40d16003-4946-4138-bf3a-320fe9605fe6","Type":"ContainerStarted","Data":"86a413fd367d9cacc3f7e61ef22174eb0104e47fef0b33c0f36c89b0c9276fbf"}
May 25 13:36:42.119836 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:42.119837    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88" event={"ID":"40d16003-4946-4138-bf3a-320fe9605fe6","Type":"ContainerStarted","Data":"fbf0ef7fe2d4c4f5d872c6bd5100f9ffb7df356d14df990e3eedcf80eb647195"}
May 25 13:36:42.120402 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:42.119853    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88"
May 25 13:36:42.135834 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:42.135787    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88" podStartSLOduration=1.135771515 podStartE2EDuration="1.135771515s" podCreationTimestamp="2026-05-25 13:36:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:36:42.134284151 +0000 UTC m=+1918.137025907" watchObservedRunningTime="2026-05-25 13:36:42.135771515 +0000 UTC m=+1918.138513272"
May 25 13:36:42.596852 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:42.596825    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa28c358-d76d-46fd-ae84-1f31f46e7fe7" path="/var/lib/kubelet/pods/aa28c358-d76d-46fd-ae84-1f31f46e7fe7/volumes"
May 25 13:36:48.130330 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:48.130301    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88"
May 25 13:36:49.027507 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:49.027464    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc" podUID="a7e53380-9d1c-42f5-bbc3-4266185be4a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
May 25 13:36:51.205346 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:51.205252    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88"]
May 25 13:36:51.205804 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:51.205478    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88" podUID="40d16003-4946-4138-bf3a-320fe9605fe6" containerName="splitter-graph-35687" containerID="cri-o://86a413fd367d9cacc3f7e61ef22174eb0104e47fef0b33c0f36c89b0c9276fbf" gracePeriod=30
May 25 13:36:51.277194 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:51.277146    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"]
May 25 13:36:51.277516 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:51.277490    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv" podUID="b0756412-f126-47cc-b3cf-fffc1e17e5be" containerName="kserve-container" containerID="cri-o://a44d95dad5cfc87ae1ebb7b8a4fe9d6f89ba1f0e99167e50c97beb749b9d3360" gracePeriod=30
May 25 13:36:51.277597 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:51.277505    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv" podUID="b0756412-f126-47cc-b3cf-fffc1e17e5be" containerName="kube-rbac-proxy" containerID="cri-o://63f35aff54159a1d9ceaf3057162f3c5b77ae1cdd72fe1db18ec65b0cae67179" gracePeriod=30
May 25 13:36:51.459673 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:51.459585    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"]
May 25 13:36:51.463255 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:51.463231    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"
May 25 13:36:51.465619 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:51.465597    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-766ab-kube-rbac-proxy-sar-config\""
May 25 13:36:51.465758 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:51.465737    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-766ab-predictor-serving-cert\""
May 25 13:36:51.474279 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:51.474234    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"]
May 25 13:36:51.539091 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:51.539043    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec4aa36e-fbc2-45e4-ae60-a072f93354c7-proxy-tls\") pod \"error-404-isvc-766ab-predictor-76978cfc8f-lfgfl\" (UID: \"ec4aa36e-fbc2-45e4-ae60-a072f93354c7\") " pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"
May 25 13:36:51.539291 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:51.539126    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r7l9\" (UniqueName: \"kubernetes.io/projected/ec4aa36e-fbc2-45e4-ae60-a072f93354c7-kube-api-access-8r7l9\") pod \"error-404-isvc-766ab-predictor-76978cfc8f-lfgfl\" (UID: \"ec4aa36e-fbc2-45e4-ae60-a072f93354c7\") " pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"
May 25 13:36:51.539291 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:51.539197    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-766ab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec4aa36e-fbc2-45e4-ae60-a072f93354c7-error-404-isvc-766ab-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-766ab-predictor-76978cfc8f-lfgfl\" (UID: \"ec4aa36e-fbc2-45e4-ae60-a072f93354c7\") " pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"
May 25 13:36:51.640263 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:51.640221    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec4aa36e-fbc2-45e4-ae60-a072f93354c7-proxy-tls\") pod \"error-404-isvc-766ab-predictor-76978cfc8f-lfgfl\" (UID: \"ec4aa36e-fbc2-45e4-ae60-a072f93354c7\") " pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"
May 25 13:36:51.640471 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:51.640307    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8r7l9\" (UniqueName: \"kubernetes.io/projected/ec4aa36e-fbc2-45e4-ae60-a072f93354c7-kube-api-access-8r7l9\") pod \"error-404-isvc-766ab-predictor-76978cfc8f-lfgfl\" (UID: \"ec4aa36e-fbc2-45e4-ae60-a072f93354c7\") " pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"
May 25 13:36:51.640471 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:51.640345    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-766ab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec4aa36e-fbc2-45e4-ae60-a072f93354c7-error-404-isvc-766ab-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-766ab-predictor-76978cfc8f-lfgfl\" (UID: \"ec4aa36e-fbc2-45e4-ae60-a072f93354c7\") " pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"
May 25 13:36:51.640471 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:36:51.640398    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-766ab-predictor-serving-cert: secret "error-404-isvc-766ab-predictor-serving-cert" not found
May 25 13:36:51.640639 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:36:51.640485    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec4aa36e-fbc2-45e4-ae60-a072f93354c7-proxy-tls podName:ec4aa36e-fbc2-45e4-ae60-a072f93354c7 nodeName:}" failed. No retries permitted until 2026-05-25 13:36:52.140461227 +0000 UTC m=+1928.143202964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ec4aa36e-fbc2-45e4-ae60-a072f93354c7-proxy-tls") pod "error-404-isvc-766ab-predictor-76978cfc8f-lfgfl" (UID: "ec4aa36e-fbc2-45e4-ae60-a072f93354c7") : secret "error-404-isvc-766ab-predictor-serving-cert" not found
May 25 13:36:51.641017 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:51.640998    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-766ab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec4aa36e-fbc2-45e4-ae60-a072f93354c7-error-404-isvc-766ab-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-766ab-predictor-76978cfc8f-lfgfl\" (UID: \"ec4aa36e-fbc2-45e4-ae60-a072f93354c7\") " pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"
May 25 13:36:51.648990 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:51.648964    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r7l9\" (UniqueName: \"kubernetes.io/projected/ec4aa36e-fbc2-45e4-ae60-a072f93354c7-kube-api-access-8r7l9\") pod \"error-404-isvc-766ab-predictor-76978cfc8f-lfgfl\" (UID: \"ec4aa36e-fbc2-45e4-ae60-a072f93354c7\") " pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"
May 25 13:36:52.144819 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:52.144774    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec4aa36e-fbc2-45e4-ae60-a072f93354c7-proxy-tls\") pod \"error-404-isvc-766ab-predictor-76978cfc8f-lfgfl\" (UID: \"ec4aa36e-fbc2-45e4-ae60-a072f93354c7\") " pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"
May 25 13:36:52.147193 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:52.147155    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec4aa36e-fbc2-45e4-ae60-a072f93354c7-proxy-tls\") pod \"error-404-isvc-766ab-predictor-76978cfc8f-lfgfl\" (UID: \"ec4aa36e-fbc2-45e4-ae60-a072f93354c7\") " pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"
May 25 13:36:52.154624 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:52.154592    2569 generic.go:358] "Generic (PLEG): container finished" podID="b0756412-f126-47cc-b3cf-fffc1e17e5be" containerID="63f35aff54159a1d9ceaf3057162f3c5b77ae1cdd72fe1db18ec65b0cae67179" exitCode=2
May 25 13:36:52.154777 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:52.154662    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv" event={"ID":"b0756412-f126-47cc-b3cf-fffc1e17e5be","Type":"ContainerDied","Data":"63f35aff54159a1d9ceaf3057162f3c5b77ae1cdd72fe1db18ec65b0cae67179"}
May 25 13:36:52.375072 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:52.375029    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"
May 25 13:36:52.509046 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:52.509016    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"]
May 25 13:36:52.511559 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:36:52.511531    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4aa36e_fbc2_45e4_ae60_a072f93354c7.slice/crio-b82471f4885ccf52d2e3faa09ce67ce59372dc74dcfe6063f17702a25e980e4a WatchSource:0}: Error finding container b82471f4885ccf52d2e3faa09ce67ce59372dc74dcfe6063f17702a25e980e4a: Status 404 returned error can't find the container with id b82471f4885ccf52d2e3faa09ce67ce59372dc74dcfe6063f17702a25e980e4a
May 25 13:36:53.128751 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:53.128712    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88" podUID="40d16003-4946-4138-bf3a-320fe9605fe6" containerName="splitter-graph-35687" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:36:53.159537 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:53.159499    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl" event={"ID":"ec4aa36e-fbc2-45e4-ae60-a072f93354c7","Type":"ContainerStarted","Data":"75d14f93ef4bf4c45d36a3781bae73c297c45fa3e9c4af7e50d8510fc402f43f"}
May 25 13:36:53.159537 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:53.159538    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl" event={"ID":"ec4aa36e-fbc2-45e4-ae60-a072f93354c7","Type":"ContainerStarted","Data":"439ad269555cc19aa7c91cec99d5b4bce9495f3b51a67b63f7a596a3f2f57a6a"}
May 25 13:36:53.159790 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:53.159548    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl" event={"ID":"ec4aa36e-fbc2-45e4-ae60-a072f93354c7","Type":"ContainerStarted","Data":"b82471f4885ccf52d2e3faa09ce67ce59372dc74dcfe6063f17702a25e980e4a"}
May 25 13:36:53.159790 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:53.159632    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"
May 25 13:36:53.179525 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:53.179470    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl" podStartSLOduration=2.179453497 podStartE2EDuration="2.179453497s" podCreationTimestamp="2026-05-25 13:36:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:36:53.177195101 +0000 UTC m=+1929.179936856" watchObservedRunningTime="2026-05-25 13:36:53.179453497 +0000 UTC m=+1929.182195251"
May 25 13:36:53.902392 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:53.902345    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv" podUID="b0756412-f126-47cc-b3cf-fffc1e17e5be" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
May 25 13:36:54.163087 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:54.163003    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"
May 25 13:36:54.164386 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:54.164361    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl" podUID="ec4aa36e-fbc2-45e4-ae60-a072f93354c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
May 25 13:36:54.821691 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:54.821669    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"
May 25 13:36:54.969364 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:54.969272    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0756412-f126-47cc-b3cf-fffc1e17e5be-proxy-tls\") pod \"b0756412-f126-47cc-b3cf-fffc1e17e5be\" (UID: \"b0756412-f126-47cc-b3cf-fffc1e17e5be\") "
May 25 13:36:54.969364 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:54.969317    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-35687-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0756412-f126-47cc-b3cf-fffc1e17e5be-error-404-isvc-35687-kube-rbac-proxy-sar-config\") pod \"b0756412-f126-47cc-b3cf-fffc1e17e5be\" (UID: \"b0756412-f126-47cc-b3cf-fffc1e17e5be\") "
May 25 13:36:54.969364 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:54.969349    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjk46\" (UniqueName: \"kubernetes.io/projected/b0756412-f126-47cc-b3cf-fffc1e17e5be-kube-api-access-xjk46\") pod \"b0756412-f126-47cc-b3cf-fffc1e17e5be\" (UID: \"b0756412-f126-47cc-b3cf-fffc1e17e5be\") "
May 25 13:36:54.969916 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:54.969690    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0756412-f126-47cc-b3cf-fffc1e17e5be-error-404-isvc-35687-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-35687-kube-rbac-proxy-sar-config") pod "b0756412-f126-47cc-b3cf-fffc1e17e5be" (UID: "b0756412-f126-47cc-b3cf-fffc1e17e5be"). InnerVolumeSpecName "error-404-isvc-35687-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:36:54.971672 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:54.971646    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0756412-f126-47cc-b3cf-fffc1e17e5be-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b0756412-f126-47cc-b3cf-fffc1e17e5be" (UID: "b0756412-f126-47cc-b3cf-fffc1e17e5be"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:36:54.971786 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:54.971655    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0756412-f126-47cc-b3cf-fffc1e17e5be-kube-api-access-xjk46" (OuterVolumeSpecName: "kube-api-access-xjk46") pod "b0756412-f126-47cc-b3cf-fffc1e17e5be" (UID: "b0756412-f126-47cc-b3cf-fffc1e17e5be"). InnerVolumeSpecName "kube-api-access-xjk46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:36:55.069943 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:55.069909    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0756412-f126-47cc-b3cf-fffc1e17e5be-proxy-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:36:55.069943 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:55.069941    2569 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-35687-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0756412-f126-47cc-b3cf-fffc1e17e5be-error-404-isvc-35687-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:36:55.069943 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:55.069952    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjk46\" (UniqueName: \"kubernetes.io/projected/b0756412-f126-47cc-b3cf-fffc1e17e5be-kube-api-access-xjk46\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:36:55.167543 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:55.167503    2569 generic.go:358] "Generic (PLEG): container finished" podID="b0756412-f126-47cc-b3cf-fffc1e17e5be" containerID="a44d95dad5cfc87ae1ebb7b8a4fe9d6f89ba1f0e99167e50c97beb749b9d3360" exitCode=0
May 25 13:36:55.167724 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:55.167571    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"
May 25 13:36:55.167724 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:55.167582    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv" event={"ID":"b0756412-f126-47cc-b3cf-fffc1e17e5be","Type":"ContainerDied","Data":"a44d95dad5cfc87ae1ebb7b8a4fe9d6f89ba1f0e99167e50c97beb749b9d3360"}
May 25 13:36:55.167724 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:55.167619    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv" event={"ID":"b0756412-f126-47cc-b3cf-fffc1e17e5be","Type":"ContainerDied","Data":"b00652429cd06f707fc3a736526fddc5ab3c31929f91cf5cdf16859b4e67bdc1"}
May 25 13:36:55.167724 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:55.167639    2569 scope.go:117] "RemoveContainer" containerID="63f35aff54159a1d9ceaf3057162f3c5b77ae1cdd72fe1db18ec65b0cae67179"
May 25 13:36:55.168201 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:55.168144    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl" podUID="ec4aa36e-fbc2-45e4-ae60-a072f93354c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
May 25 13:36:55.176011 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:55.175989    2569 scope.go:117] "RemoveContainer" containerID="a44d95dad5cfc87ae1ebb7b8a4fe9d6f89ba1f0e99167e50c97beb749b9d3360"
May 25 13:36:55.183866 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:55.183846    2569 scope.go:117] "RemoveContainer" containerID="63f35aff54159a1d9ceaf3057162f3c5b77ae1cdd72fe1db18ec65b0cae67179"
May 25 13:36:55.184131 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:36:55.184111    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63f35aff54159a1d9ceaf3057162f3c5b77ae1cdd72fe1db18ec65b0cae67179\": container with ID starting with 63f35aff54159a1d9ceaf3057162f3c5b77ae1cdd72fe1db18ec65b0cae67179 not found: ID does not exist" containerID="63f35aff54159a1d9ceaf3057162f3c5b77ae1cdd72fe1db18ec65b0cae67179"
May 25 13:36:55.184228 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:55.184144    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63f35aff54159a1d9ceaf3057162f3c5b77ae1cdd72fe1db18ec65b0cae67179"} err="failed to get container status \"63f35aff54159a1d9ceaf3057162f3c5b77ae1cdd72fe1db18ec65b0cae67179\": rpc error: code = NotFound desc = could not find container \"63f35aff54159a1d9ceaf3057162f3c5b77ae1cdd72fe1db18ec65b0cae67179\": container with ID starting with 63f35aff54159a1d9ceaf3057162f3c5b77ae1cdd72fe1db18ec65b0cae67179 not found: ID does not exist"
May 25 13:36:55.184228 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:55.184201    2569 scope.go:117] "RemoveContainer" containerID="a44d95dad5cfc87ae1ebb7b8a4fe9d6f89ba1f0e99167e50c97beb749b9d3360"
May 25 13:36:55.184437 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:36:55.184416    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a44d95dad5cfc87ae1ebb7b8a4fe9d6f89ba1f0e99167e50c97beb749b9d3360\": container with ID starting with a44d95dad5cfc87ae1ebb7b8a4fe9d6f89ba1f0e99167e50c97beb749b9d3360 not found: ID does not exist" containerID="a44d95dad5cfc87ae1ebb7b8a4fe9d6f89ba1f0e99167e50c97beb749b9d3360"
May 25 13:36:55.184538 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:55.184441    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a44d95dad5cfc87ae1ebb7b8a4fe9d6f89ba1f0e99167e50c97beb749b9d3360"} err="failed to get container status \"a44d95dad5cfc87ae1ebb7b8a4fe9d6f89ba1f0e99167e50c97beb749b9d3360\": rpc error: code = NotFound desc = could not find container \"a44d95dad5cfc87ae1ebb7b8a4fe9d6f89ba1f0e99167e50c97beb749b9d3360\": container with ID starting with a44d95dad5cfc87ae1ebb7b8a4fe9d6f89ba1f0e99167e50c97beb749b9d3360 not found: ID does not exist"
May 25 13:36:55.189861 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:55.189838    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"]
May 25 13:36:55.195830 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:55.195806    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-35687-predictor-d8849b54c-bcggv"]
May 25 13:36:56.597087 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:56.597049    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0756412-f126-47cc-b3cf-fffc1e17e5be" path="/var/lib/kubelet/pods/b0756412-f126-47cc-b3cf-fffc1e17e5be/volumes"
May 25 13:36:58.130035 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:58.129995    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88" podUID="40d16003-4946-4138-bf3a-320fe9605fe6" containerName="splitter-graph-35687" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:36:59.027266 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:36:59.027233    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"
May 25 13:37:00.173581 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:00.173552    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"
May 25 13:37:00.174118 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:00.174093    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl" podUID="ec4aa36e-fbc2-45e4-ae60-a072f93354c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
May 25 13:37:03.128982 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:03.128943    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88" podUID="40d16003-4946-4138-bf3a-320fe9605fe6" containerName="splitter-graph-35687" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:37:03.129413 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:03.129060    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88"
May 25 13:37:08.128822 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:08.128778    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88" podUID="40d16003-4946-4138-bf3a-320fe9605fe6" containerName="splitter-graph-35687" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:37:10.174621 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:10.174580    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl" podUID="ec4aa36e-fbc2-45e4-ae60-a072f93354c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
May 25 13:37:11.159433 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:11.159395    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq"]
May 25 13:37:11.159855 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:11.159836    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0756412-f126-47cc-b3cf-fffc1e17e5be" containerName="kube-rbac-proxy"
May 25 13:37:11.159936 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:11.159857    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0756412-f126-47cc-b3cf-fffc1e17e5be" containerName="kube-rbac-proxy"
May 25 13:37:11.159936 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:11.159900    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0756412-f126-47cc-b3cf-fffc1e17e5be" containerName="kserve-container"
May 25 13:37:11.159936 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:11.159910    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0756412-f126-47cc-b3cf-fffc1e17e5be" containerName="kserve-container"
May 25 13:37:11.160097 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:11.159998    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="b0756412-f126-47cc-b3cf-fffc1e17e5be" containerName="kserve-container"
May 25 13:37:11.160097 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:11.160019    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="b0756412-f126-47cc-b3cf-fffc1e17e5be" containerName="kube-rbac-proxy"
May 25 13:37:11.164667 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:11.164646    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq"
May 25 13:37:11.168345 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:11.168326    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-12313-kube-rbac-proxy-sar-config\""
May 25 13:37:11.168345 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:11.168334    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-12313-serving-cert\""
May 25 13:37:11.178535 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:11.178503    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq"]
May 25 13:37:11.201506 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:11.201470    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/915f615d-8e11-4584-bb10-57733fba7e83-proxy-tls\") pod \"switch-graph-12313-76b55b9df6-96qlq\" (UID: \"915f615d-8e11-4584-bb10-57733fba7e83\") " pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq"
May 25 13:37:11.201690 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:11.201515    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/915f615d-8e11-4584-bb10-57733fba7e83-openshift-service-ca-bundle\") pod \"switch-graph-12313-76b55b9df6-96qlq\" (UID: \"915f615d-8e11-4584-bb10-57733fba7e83\") " pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq"
May 25 13:37:11.302708 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:11.302669    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/915f615d-8e11-4584-bb10-57733fba7e83-openshift-service-ca-bundle\") pod \"switch-graph-12313-76b55b9df6-96qlq\" (UID: \"915f615d-8e11-4584-bb10-57733fba7e83\") " pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq"
May 25 13:37:11.302886 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:11.302753    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/915f615d-8e11-4584-bb10-57733fba7e83-proxy-tls\") pod \"switch-graph-12313-76b55b9df6-96qlq\" (UID: \"915f615d-8e11-4584-bb10-57733fba7e83\") " pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq"
May 25 13:37:11.303353 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:11.303325    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/915f615d-8e11-4584-bb10-57733fba7e83-openshift-service-ca-bundle\") pod \"switch-graph-12313-76b55b9df6-96qlq\" (UID: \"915f615d-8e11-4584-bb10-57733fba7e83\") " pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq"
May 25 13:37:11.305120 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:11.305099    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/915f615d-8e11-4584-bb10-57733fba7e83-proxy-tls\") pod \"switch-graph-12313-76b55b9df6-96qlq\" (UID: \"915f615d-8e11-4584-bb10-57733fba7e83\") " pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq"
May 25 13:37:11.479961 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:11.479857    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq"
May 25 13:37:11.608346 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:11.608314    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq"]
May 25 13:37:11.611833 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:37:11.611801    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915f615d_8e11_4584_bb10_57733fba7e83.slice/crio-6a6abadd33c342807ef2f869eac8bffd9a7cd44dd7de574c0f8969ddc63756ee WatchSource:0}: Error finding container 6a6abadd33c342807ef2f869eac8bffd9a7cd44dd7de574c0f8969ddc63756ee: Status 404 returned error can't find the container with id 6a6abadd33c342807ef2f869eac8bffd9a7cd44dd7de574c0f8969ddc63756ee
May 25 13:37:12.231056 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:12.231014    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq" event={"ID":"915f615d-8e11-4584-bb10-57733fba7e83","Type":"ContainerStarted","Data":"a964ecb83ff75ae2870ed3bfa129e0755b96eb9799456c2b25f6667c45da2f37"}
May 25 13:37:12.231444 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:12.231062    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq" event={"ID":"915f615d-8e11-4584-bb10-57733fba7e83","Type":"ContainerStarted","Data":"6a6abadd33c342807ef2f869eac8bffd9a7cd44dd7de574c0f8969ddc63756ee"}
May 25 13:37:12.231444 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:12.231097    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq"
May 25 13:37:12.246572 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:12.246512    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq" podStartSLOduration=1.246495787 podStartE2EDuration="1.246495787s" podCreationTimestamp="2026-05-25 13:37:11 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:37:12.246048044 +0000 UTC m=+1948.248789798" watchObservedRunningTime="2026-05-25 13:37:12.246495787 +0000 UTC m=+1948.249237542"
May 25 13:37:13.128273 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:13.128236    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88" podUID="40d16003-4946-4138-bf3a-320fe9605fe6" containerName="splitter-graph-35687" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:37:18.128266 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:18.128220    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88" podUID="40d16003-4946-4138-bf3a-320fe9605fe6" containerName="splitter-graph-35687" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:37:18.239697 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:18.239666    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq"
May 25 13:37:20.174266 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:20.174224    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl" podUID="ec4aa36e-fbc2-45e4-ae60-a072f93354c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
May 25 13:37:21.260269 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:21.260236    2569 generic.go:358] "Generic (PLEG): container finished" podID="40d16003-4946-4138-bf3a-320fe9605fe6" containerID="86a413fd367d9cacc3f7e61ef22174eb0104e47fef0b33c0f36c89b0c9276fbf" exitCode=0
May 25 13:37:21.260645 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:21.260320    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88" event={"ID":"40d16003-4946-4138-bf3a-320fe9605fe6","Type":"ContainerDied","Data":"86a413fd367d9cacc3f7e61ef22174eb0104e47fef0b33c0f36c89b0c9276fbf"}
May 25 13:37:21.357009 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:21.356983    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88"
May 25 13:37:21.393450 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:21.393411    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40d16003-4946-4138-bf3a-320fe9605fe6-openshift-service-ca-bundle\") pod \"40d16003-4946-4138-bf3a-320fe9605fe6\" (UID: \"40d16003-4946-4138-bf3a-320fe9605fe6\") "
May 25 13:37:21.393637 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:21.393469    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40d16003-4946-4138-bf3a-320fe9605fe6-proxy-tls\") pod \"40d16003-4946-4138-bf3a-320fe9605fe6\" (UID: \"40d16003-4946-4138-bf3a-320fe9605fe6\") "
May 25 13:37:21.393874 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:21.393846    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d16003-4946-4138-bf3a-320fe9605fe6-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "40d16003-4946-4138-bf3a-320fe9605fe6" (UID: "40d16003-4946-4138-bf3a-320fe9605fe6"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:37:21.395548 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:21.395527    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40d16003-4946-4138-bf3a-320fe9605fe6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "40d16003-4946-4138-bf3a-320fe9605fe6" (UID: "40d16003-4946-4138-bf3a-320fe9605fe6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:37:21.494513 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:21.494425    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40d16003-4946-4138-bf3a-320fe9605fe6-openshift-service-ca-bundle\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:37:21.494513 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:21.494459    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40d16003-4946-4138-bf3a-320fe9605fe6-proxy-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:37:22.265375 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:22.265338    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88" event={"ID":"40d16003-4946-4138-bf3a-320fe9605fe6","Type":"ContainerDied","Data":"fbf0ef7fe2d4c4f5d872c6bd5100f9ffb7df356d14df990e3eedcf80eb647195"}
May 25 13:37:22.265845 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:22.265388    2569 scope.go:117] "RemoveContainer" containerID="86a413fd367d9cacc3f7e61ef22174eb0104e47fef0b33c0f36c89b0c9276fbf"
May 25 13:37:22.265845 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:22.265351    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88"
May 25 13:37:22.285754 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:22.285724    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88"]
May 25 13:37:22.289489 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:22.289466    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-35687-79cdffcd67-wpm88"]
May 25 13:37:22.597279 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:22.597250    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40d16003-4946-4138-bf3a-320fe9605fe6" path="/var/lib/kubelet/pods/40d16003-4946-4138-bf3a-320fe9605fe6/volumes"
May 25 13:37:30.174447 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:30.174399    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl" podUID="ec4aa36e-fbc2-45e4-ae60-a072f93354c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
May 25 13:37:40.175468 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:40.175379    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"
May 25 13:37:51.433195 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:51.433136    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6"]
May 25 13:37:51.433627 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:51.433526    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d16003-4946-4138-bf3a-320fe9605fe6" containerName="splitter-graph-35687"
May 25 13:37:51.433627 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:51.433540    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d16003-4946-4138-bf3a-320fe9605fe6" containerName="splitter-graph-35687"
May 25 13:37:51.433627 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:51.433607    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="40d16003-4946-4138-bf3a-320fe9605fe6" containerName="splitter-graph-35687"
May 25 13:37:51.436522 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:51.436505    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6"
May 25 13:37:51.439389 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:51.439361    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-766ab-serving-cert\""
May 25 13:37:51.439521 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:51.439404    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-766ab-kube-rbac-proxy-sar-config\""
May 25 13:37:51.446125 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:51.446103    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6"]
May 25 13:37:51.552427 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:51.552384    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bec9eb96-a8e6-4b65-83df-5f49519d5235-proxy-tls\") pod \"splitter-graph-766ab-98578b9d8-6bwk6\" (UID: \"bec9eb96-a8e6-4b65-83df-5f49519d5235\") " pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6"
May 25 13:37:51.552647 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:51.552524    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bec9eb96-a8e6-4b65-83df-5f49519d5235-openshift-service-ca-bundle\") pod \"splitter-graph-766ab-98578b9d8-6bwk6\" (UID: \"bec9eb96-a8e6-4b65-83df-5f49519d5235\") " pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6"
May 25 13:37:51.653422 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:51.653381    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bec9eb96-a8e6-4b65-83df-5f49519d5235-openshift-service-ca-bundle\") pod \"splitter-graph-766ab-98578b9d8-6bwk6\" (UID: \"bec9eb96-a8e6-4b65-83df-5f49519d5235\") " pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6"
May 25 13:37:51.653611 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:51.653455    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bec9eb96-a8e6-4b65-83df-5f49519d5235-proxy-tls\") pod \"splitter-graph-766ab-98578b9d8-6bwk6\" (UID: \"bec9eb96-a8e6-4b65-83df-5f49519d5235\") " pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6"
May 25 13:37:51.654021 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:51.654000    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bec9eb96-a8e6-4b65-83df-5f49519d5235-openshift-service-ca-bundle\") pod \"splitter-graph-766ab-98578b9d8-6bwk6\" (UID: \"bec9eb96-a8e6-4b65-83df-5f49519d5235\") " pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6"
May 25 13:37:51.655962 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:51.655937    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bec9eb96-a8e6-4b65-83df-5f49519d5235-proxy-tls\") pod \"splitter-graph-766ab-98578b9d8-6bwk6\" (UID: \"bec9eb96-a8e6-4b65-83df-5f49519d5235\") " pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6"
May 25 13:37:51.748297 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:51.748207    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6"
May 25 13:37:51.874857 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:51.874824    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6"]
May 25 13:37:51.878056 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:37:51.878014    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec9eb96_a8e6_4b65_83df_5f49519d5235.slice/crio-62752e0c33160ce61cab1e67f8136d53959128b512518aef0bfc408cd9e253ff WatchSource:0}: Error finding container 62752e0c33160ce61cab1e67f8136d53959128b512518aef0bfc408cd9e253ff: Status 404 returned error can't find the container with id 62752e0c33160ce61cab1e67f8136d53959128b512518aef0bfc408cd9e253ff
May 25 13:37:52.377605 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:52.377571    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6" event={"ID":"bec9eb96-a8e6-4b65-83df-5f49519d5235","Type":"ContainerStarted","Data":"6b5632ab0ec85725f9994ed0d60d148082d378d508ca3e7b41345c18b1697489"}
May 25 13:37:52.377605 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:52.377607    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6" event={"ID":"bec9eb96-a8e6-4b65-83df-5f49519d5235","Type":"ContainerStarted","Data":"62752e0c33160ce61cab1e67f8136d53959128b512518aef0bfc408cd9e253ff"}
May 25 13:37:52.377838 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:52.377686    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6"
May 25 13:37:52.395460 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:52.395408    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6" podStartSLOduration=1.3953923590000001 podStartE2EDuration="1.395392359s" podCreationTimestamp="2026-05-25 13:37:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:37:52.393575526 +0000 UTC m=+1988.396317280" watchObservedRunningTime="2026-05-25 13:37:52.395392359 +0000 UTC m=+1988.398134113"
May 25 13:37:58.387512 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:37:58.387479    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6"
May 25 13:39:45.677532 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:39:45.677502    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/1.log"
May 25 13:39:45.681338 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:39:45.681315    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/1.log"
May 25 13:39:45.682622 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:39:45.682602    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:39:45.686184 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:39:45.686156    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:44:45.704214 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:44:45.704076    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/1.log"
May 25 13:44:45.707465 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:44:45.707307    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/1.log"
May 25 13:44:45.711799 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:44:45.711776    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:44:45.714320 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:44:45.714304    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:46:06.182372 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:06.182339    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6"]
May 25 13:46:06.182899 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:06.182577    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6" podUID="bec9eb96-a8e6-4b65-83df-5f49519d5235" containerName="splitter-graph-766ab" containerID="cri-o://6b5632ab0ec85725f9994ed0d60d148082d378d508ca3e7b41345c18b1697489" gracePeriod=30
May 25 13:46:06.312554 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:06.312521    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"]
May 25 13:46:06.312873 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:06.312848    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl" podUID="ec4aa36e-fbc2-45e4-ae60-a072f93354c7" containerName="kserve-container" containerID="cri-o://439ad269555cc19aa7c91cec99d5b4bce9495f3b51a67b63f7a596a3f2f57a6a" gracePeriod=30
May 25 13:46:06.312972 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:06.312892    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl" podUID="ec4aa36e-fbc2-45e4-ae60-a072f93354c7" containerName="kube-rbac-proxy" containerID="cri-o://75d14f93ef4bf4c45d36a3781bae73c297c45fa3e9c4af7e50d8510fc402f43f" gracePeriod=30
May 25 13:46:07.062397 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:07.062364    2569 generic.go:358] "Generic (PLEG): container finished" podID="ec4aa36e-fbc2-45e4-ae60-a072f93354c7" containerID="75d14f93ef4bf4c45d36a3781bae73c297c45fa3e9c4af7e50d8510fc402f43f" exitCode=2
May 25 13:46:07.062575 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:07.062437    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl" event={"ID":"ec4aa36e-fbc2-45e4-ae60-a072f93354c7","Type":"ContainerDied","Data":"75d14f93ef4bf4c45d36a3781bae73c297c45fa3e9c4af7e50d8510fc402f43f"}
May 25 13:46:08.386050 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:08.386006    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6" podUID="bec9eb96-a8e6-4b65-83df-5f49519d5235" containerName="splitter-graph-766ab" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:46:09.658237 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:09.658212    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"
May 25 13:46:09.786369 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:09.786287    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r7l9\" (UniqueName: \"kubernetes.io/projected/ec4aa36e-fbc2-45e4-ae60-a072f93354c7-kube-api-access-8r7l9\") pod \"ec4aa36e-fbc2-45e4-ae60-a072f93354c7\" (UID: \"ec4aa36e-fbc2-45e4-ae60-a072f93354c7\") "
May 25 13:46:09.786369 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:09.786333    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-766ab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec4aa36e-fbc2-45e4-ae60-a072f93354c7-error-404-isvc-766ab-kube-rbac-proxy-sar-config\") pod \"ec4aa36e-fbc2-45e4-ae60-a072f93354c7\" (UID: \"ec4aa36e-fbc2-45e4-ae60-a072f93354c7\") "
May 25 13:46:09.786552 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:09.786374    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec4aa36e-fbc2-45e4-ae60-a072f93354c7-proxy-tls\") pod \"ec4aa36e-fbc2-45e4-ae60-a072f93354c7\" (UID: \"ec4aa36e-fbc2-45e4-ae60-a072f93354c7\") "
May 25 13:46:09.786754 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:09.786725    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4aa36e-fbc2-45e4-ae60-a072f93354c7-error-404-isvc-766ab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-766ab-kube-rbac-proxy-sar-config") pod "ec4aa36e-fbc2-45e4-ae60-a072f93354c7" (UID: "ec4aa36e-fbc2-45e4-ae60-a072f93354c7"). InnerVolumeSpecName "error-404-isvc-766ab-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:46:09.788488 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:09.788461    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4aa36e-fbc2-45e4-ae60-a072f93354c7-kube-api-access-8r7l9" (OuterVolumeSpecName: "kube-api-access-8r7l9") pod "ec4aa36e-fbc2-45e4-ae60-a072f93354c7" (UID: "ec4aa36e-fbc2-45e4-ae60-a072f93354c7"). InnerVolumeSpecName "kube-api-access-8r7l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:46:09.788599 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:09.788491    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec4aa36e-fbc2-45e4-ae60-a072f93354c7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ec4aa36e-fbc2-45e4-ae60-a072f93354c7" (UID: "ec4aa36e-fbc2-45e4-ae60-a072f93354c7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:46:09.887120 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:09.887083    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r7l9\" (UniqueName: \"kubernetes.io/projected/ec4aa36e-fbc2-45e4-ae60-a072f93354c7-kube-api-access-8r7l9\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:46:09.887120 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:09.887117    2569 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-766ab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec4aa36e-fbc2-45e4-ae60-a072f93354c7-error-404-isvc-766ab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:46:09.887120 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:09.887128    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec4aa36e-fbc2-45e4-ae60-a072f93354c7-proxy-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:46:10.075325 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:10.075220    2569 generic.go:358] "Generic (PLEG): container finished" podID="ec4aa36e-fbc2-45e4-ae60-a072f93354c7" containerID="439ad269555cc19aa7c91cec99d5b4bce9495f3b51a67b63f7a596a3f2f57a6a" exitCode=0
May 25 13:46:10.075325 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:10.075301    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"
May 25 13:46:10.075325 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:10.075307    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl" event={"ID":"ec4aa36e-fbc2-45e4-ae60-a072f93354c7","Type":"ContainerDied","Data":"439ad269555cc19aa7c91cec99d5b4bce9495f3b51a67b63f7a596a3f2f57a6a"}
May 25 13:46:10.075606 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:10.075347    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl" event={"ID":"ec4aa36e-fbc2-45e4-ae60-a072f93354c7","Type":"ContainerDied","Data":"b82471f4885ccf52d2e3faa09ce67ce59372dc74dcfe6063f17702a25e980e4a"}
May 25 13:46:10.075606 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:10.075369    2569 scope.go:117] "RemoveContainer" containerID="75d14f93ef4bf4c45d36a3781bae73c297c45fa3e9c4af7e50d8510fc402f43f"
May 25 13:46:10.083765 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:10.083744    2569 scope.go:117] "RemoveContainer" containerID="439ad269555cc19aa7c91cec99d5b4bce9495f3b51a67b63f7a596a3f2f57a6a"
May 25 13:46:10.092335 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:10.092310    2569 scope.go:117] "RemoveContainer" containerID="75d14f93ef4bf4c45d36a3781bae73c297c45fa3e9c4af7e50d8510fc402f43f"
May 25 13:46:10.092598 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:46:10.092581    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d14f93ef4bf4c45d36a3781bae73c297c45fa3e9c4af7e50d8510fc402f43f\": container with ID starting with 75d14f93ef4bf4c45d36a3781bae73c297c45fa3e9c4af7e50d8510fc402f43f not found: ID does not exist" containerID="75d14f93ef4bf4c45d36a3781bae73c297c45fa3e9c4af7e50d8510fc402f43f"
May 25 13:46:10.092659 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:10.092606    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d14f93ef4bf4c45d36a3781bae73c297c45fa3e9c4af7e50d8510fc402f43f"} err="failed to get container status \"75d14f93ef4bf4c45d36a3781bae73c297c45fa3e9c4af7e50d8510fc402f43f\": rpc error: code = NotFound desc = could not find container \"75d14f93ef4bf4c45d36a3781bae73c297c45fa3e9c4af7e50d8510fc402f43f\": container with ID starting with 75d14f93ef4bf4c45d36a3781bae73c297c45fa3e9c4af7e50d8510fc402f43f not found: ID does not exist"
May 25 13:46:10.092659 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:10.092624    2569 scope.go:117] "RemoveContainer" containerID="439ad269555cc19aa7c91cec99d5b4bce9495f3b51a67b63f7a596a3f2f57a6a"
May 25 13:46:10.092862 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:46:10.092841    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"439ad269555cc19aa7c91cec99d5b4bce9495f3b51a67b63f7a596a3f2f57a6a\": container with ID starting with 439ad269555cc19aa7c91cec99d5b4bce9495f3b51a67b63f7a596a3f2f57a6a not found: ID does not exist" containerID="439ad269555cc19aa7c91cec99d5b4bce9495f3b51a67b63f7a596a3f2f57a6a"
May 25 13:46:10.092935 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:10.092874    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"439ad269555cc19aa7c91cec99d5b4bce9495f3b51a67b63f7a596a3f2f57a6a"} err="failed to get container status \"439ad269555cc19aa7c91cec99d5b4bce9495f3b51a67b63f7a596a3f2f57a6a\": rpc error: code = NotFound desc = could not find container \"439ad269555cc19aa7c91cec99d5b4bce9495f3b51a67b63f7a596a3f2f57a6a\": container with ID starting with 439ad269555cc19aa7c91cec99d5b4bce9495f3b51a67b63f7a596a3f2f57a6a not found: ID does not exist"
May 25 13:46:10.096131 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:10.096109    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"]
May 25 13:46:10.099921 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:10.099898    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-766ab-predictor-76978cfc8f-lfgfl"]
May 25 13:46:10.597489 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:10.597439    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec4aa36e-fbc2-45e4-ae60-a072f93354c7" path="/var/lib/kubelet/pods/ec4aa36e-fbc2-45e4-ae60-a072f93354c7/volumes"
May 25 13:46:13.386337 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:13.386298    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6" podUID="bec9eb96-a8e6-4b65-83df-5f49519d5235" containerName="splitter-graph-766ab" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:46:18.385570 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:18.385524    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6" podUID="bec9eb96-a8e6-4b65-83df-5f49519d5235" containerName="splitter-graph-766ab" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:46:18.386003 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:18.385645    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6"
May 25 13:46:23.385256 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:23.385214    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6" podUID="bec9eb96-a8e6-4b65-83df-5f49519d5235" containerName="splitter-graph-766ab" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:46:28.385329 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:28.385285    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6" podUID="bec9eb96-a8e6-4b65-83df-5f49519d5235" containerName="splitter-graph-766ab" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:46:33.385417 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:33.385373    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6" podUID="bec9eb96-a8e6-4b65-83df-5f49519d5235" containerName="splitter-graph-766ab" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:46:36.332289 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:36.332257    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6"
May 25 13:46:36.420392 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:36.420356    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bec9eb96-a8e6-4b65-83df-5f49519d5235-proxy-tls\") pod \"bec9eb96-a8e6-4b65-83df-5f49519d5235\" (UID: \"bec9eb96-a8e6-4b65-83df-5f49519d5235\") "
May 25 13:46:36.420392 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:36.420397    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bec9eb96-a8e6-4b65-83df-5f49519d5235-openshift-service-ca-bundle\") pod \"bec9eb96-a8e6-4b65-83df-5f49519d5235\" (UID: \"bec9eb96-a8e6-4b65-83df-5f49519d5235\") "
May 25 13:46:36.420784 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:36.420760    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bec9eb96-a8e6-4b65-83df-5f49519d5235-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "bec9eb96-a8e6-4b65-83df-5f49519d5235" (UID: "bec9eb96-a8e6-4b65-83df-5f49519d5235"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:46:36.422567 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:36.422542    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bec9eb96-a8e6-4b65-83df-5f49519d5235-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bec9eb96-a8e6-4b65-83df-5f49519d5235" (UID: "bec9eb96-a8e6-4b65-83df-5f49519d5235"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:46:36.521451 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:36.521353    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bec9eb96-a8e6-4b65-83df-5f49519d5235-proxy-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:46:36.521451 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:36.521389    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bec9eb96-a8e6-4b65-83df-5f49519d5235-openshift-service-ca-bundle\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:46:37.167000 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:37.166963    2569 generic.go:358] "Generic (PLEG): container finished" podID="bec9eb96-a8e6-4b65-83df-5f49519d5235" containerID="6b5632ab0ec85725f9994ed0d60d148082d378d508ca3e7b41345c18b1697489" exitCode=0
May 25 13:46:37.167243 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:37.167023    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6"
May 25 13:46:37.167243 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:37.167054    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6" event={"ID":"bec9eb96-a8e6-4b65-83df-5f49519d5235","Type":"ContainerDied","Data":"6b5632ab0ec85725f9994ed0d60d148082d378d508ca3e7b41345c18b1697489"}
May 25 13:46:37.167243 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:37.167091    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6" event={"ID":"bec9eb96-a8e6-4b65-83df-5f49519d5235","Type":"ContainerDied","Data":"62752e0c33160ce61cab1e67f8136d53959128b512518aef0bfc408cd9e253ff"}
May 25 13:46:37.167243 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:37.167106    2569 scope.go:117] "RemoveContainer" containerID="6b5632ab0ec85725f9994ed0d60d148082d378d508ca3e7b41345c18b1697489"
May 25 13:46:37.175299 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:37.175279    2569 scope.go:117] "RemoveContainer" containerID="6b5632ab0ec85725f9994ed0d60d148082d378d508ca3e7b41345c18b1697489"
May 25 13:46:37.175541 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:46:37.175522    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b5632ab0ec85725f9994ed0d60d148082d378d508ca3e7b41345c18b1697489\": container with ID starting with 6b5632ab0ec85725f9994ed0d60d148082d378d508ca3e7b41345c18b1697489 not found: ID does not exist" containerID="6b5632ab0ec85725f9994ed0d60d148082d378d508ca3e7b41345c18b1697489"
May 25 13:46:37.175592 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:37.175550    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b5632ab0ec85725f9994ed0d60d148082d378d508ca3e7b41345c18b1697489"} err="failed to get container status \"6b5632ab0ec85725f9994ed0d60d148082d378d508ca3e7b41345c18b1697489\": rpc error: code = NotFound desc = could not find container \"6b5632ab0ec85725f9994ed0d60d148082d378d508ca3e7b41345c18b1697489\": container with ID starting with 6b5632ab0ec85725f9994ed0d60d148082d378d508ca3e7b41345c18b1697489 not found: ID does not exist"
May 25 13:46:37.182826 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:37.182804    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6"]
May 25 13:46:37.185950 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:37.185927    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-766ab-98578b9d8-6bwk6"]
May 25 13:46:38.597403 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:46:38.597373    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bec9eb96-a8e6-4b65-83df-5f49519d5235" path="/var/lib/kubelet/pods/bec9eb96-a8e6-4b65-83df-5f49519d5235/volumes"
May 25 13:49:45.730669 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:49:45.730551    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/1.log"
May 25 13:49:45.733904 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:49:45.733394    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/1.log"
May 25 13:49:45.735761 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:49:45.735734    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:49:45.739022 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:49:45.739004    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:53:30.575506 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:30.575459    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq"]
May 25 13:53:30.576053 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:30.575758    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq" podUID="915f615d-8e11-4584-bb10-57733fba7e83" containerName="switch-graph-12313" containerID="cri-o://a964ecb83ff75ae2870ed3bfa129e0755b96eb9799456c2b25f6667c45da2f37" gracePeriod=30
May 25 13:53:30.757915 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:30.757883    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"]
May 25 13:53:30.758249 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:30.758192    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc" podUID="a7e53380-9d1c-42f5-bbc3-4266185be4a0" containerName="kserve-container" containerID="cri-o://65eed7ee6ce35f4bd51ba461b1bc93f48cefc31b8a179ec35dd835d249b8d35c" gracePeriod=30
May 25 13:53:30.758398 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:30.758234    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc" podUID="a7e53380-9d1c-42f5-bbc3-4266185be4a0" containerName="kube-rbac-proxy" containerID="cri-o://6e9d5cc18fbe7d13369784000fe9f8d97c3be253e48424a9990702a3204b1c9b" gracePeriod=30
May 25 13:53:31.444347 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.444310    2569 generic.go:358] "Generic (PLEG): container finished" podID="a7e53380-9d1c-42f5-bbc3-4266185be4a0" containerID="6e9d5cc18fbe7d13369784000fe9f8d97c3be253e48424a9990702a3204b1c9b" exitCode=2
May 25 13:53:31.444518 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.444373    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc" event={"ID":"a7e53380-9d1c-42f5-bbc3-4266185be4a0","Type":"ContainerDied","Data":"6e9d5cc18fbe7d13369784000fe9f8d97c3be253e48424a9990702a3204b1c9b"}
May 25 13:53:31.742410 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.742324    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bg2bw/must-gather-qgxtt"]
May 25 13:53:31.742860 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.742840    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec4aa36e-fbc2-45e4-ae60-a072f93354c7" containerName="kserve-container"
May 25 13:53:31.742936 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.742864    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec4aa36e-fbc2-45e4-ae60-a072f93354c7" containerName="kserve-container"
May 25 13:53:31.742936 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.742897    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bec9eb96-a8e6-4b65-83df-5f49519d5235" containerName="splitter-graph-766ab"
May 25 13:53:31.742936 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.742908    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bec9eb96-a8e6-4b65-83df-5f49519d5235" containerName="splitter-graph-766ab"
May 25 13:53:31.742936 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.742929    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec4aa36e-fbc2-45e4-ae60-a072f93354c7" containerName="kube-rbac-proxy"
May 25 13:53:31.743138 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.742938    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec4aa36e-fbc2-45e4-ae60-a072f93354c7" containerName="kube-rbac-proxy"
May 25 13:53:31.743138 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.743026    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="ec4aa36e-fbc2-45e4-ae60-a072f93354c7" containerName="kube-rbac-proxy"
May 25 13:53:31.743138 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.743041    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="bec9eb96-a8e6-4b65-83df-5f49519d5235" containerName="splitter-graph-766ab"
May 25 13:53:31.743138 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.743053    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="ec4aa36e-fbc2-45e4-ae60-a072f93354c7" containerName="kserve-container"
May 25 13:53:31.746264 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.746242    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bg2bw/must-gather-qgxtt"
May 25 13:53:31.750751 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.750727    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bg2bw\"/\"default-dockercfg-knkfm\""
May 25 13:53:31.751710 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.751684    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bg2bw\"/\"kube-root-ca.crt\""
May 25 13:53:31.751834 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.751689    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bg2bw\"/\"openshift-service-ca.crt\""
May 25 13:53:31.768382 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.768354    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bg2bw/must-gather-qgxtt"]
May 25 13:53:31.887783 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.887751    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f722ffe9-8cfd-4e79-969c-31985090971a-must-gather-output\") pod \"must-gather-qgxtt\" (UID: \"f722ffe9-8cfd-4e79-969c-31985090971a\") " pod="openshift-must-gather-bg2bw/must-gather-qgxtt"
May 25 13:53:31.887976 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.887817    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpcnz\" (UniqueName: \"kubernetes.io/projected/f722ffe9-8cfd-4e79-969c-31985090971a-kube-api-access-qpcnz\") pod \"must-gather-qgxtt\" (UID: \"f722ffe9-8cfd-4e79-969c-31985090971a\") " pod="openshift-must-gather-bg2bw/must-gather-qgxtt"
May 25 13:53:31.988872 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.988836    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qpcnz\" (UniqueName: \"kubernetes.io/projected/f722ffe9-8cfd-4e79-969c-31985090971a-kube-api-access-qpcnz\") pod \"must-gather-qgxtt\" (UID: \"f722ffe9-8cfd-4e79-969c-31985090971a\") " pod="openshift-must-gather-bg2bw/must-gather-qgxtt"
May 25 13:53:31.989094 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.988942    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f722ffe9-8cfd-4e79-969c-31985090971a-must-gather-output\") pod \"must-gather-qgxtt\" (UID: \"f722ffe9-8cfd-4e79-969c-31985090971a\") " pod="openshift-must-gather-bg2bw/must-gather-qgxtt"
May 25 13:53:31.989328 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.989308    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f722ffe9-8cfd-4e79-969c-31985090971a-must-gather-output\") pod \"must-gather-qgxtt\" (UID: \"f722ffe9-8cfd-4e79-969c-31985090971a\") " pod="openshift-must-gather-bg2bw/must-gather-qgxtt"
May 25 13:53:31.997071 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:31.997009    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpcnz\" (UniqueName: \"kubernetes.io/projected/f722ffe9-8cfd-4e79-969c-31985090971a-kube-api-access-qpcnz\") pod \"must-gather-qgxtt\" (UID: \"f722ffe9-8cfd-4e79-969c-31985090971a\") " pod="openshift-must-gather-bg2bw/must-gather-qgxtt"
May 25 13:53:32.069074 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:32.069037    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bg2bw/must-gather-qgxtt"
May 25 13:53:32.201298 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:32.201263    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bg2bw/must-gather-qgxtt"]
May 25 13:53:32.204296 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:53:32.204269    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf722ffe9_8cfd_4e79_969c_31985090971a.slice/crio-62f9e32c3e745e8e16febd819c7f69acdb4e125640c40043b3eebb4e669a3dba WatchSource:0}: Error finding container 62f9e32c3e745e8e16febd819c7f69acdb4e125640c40043b3eebb4e669a3dba: Status 404 returned error can't find the container with id 62f9e32c3e745e8e16febd819c7f69acdb4e125640c40043b3eebb4e669a3dba
May 25 13:53:32.206012 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:32.205993    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:53:32.448931 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:32.448894    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bg2bw/must-gather-qgxtt" event={"ID":"f722ffe9-8cfd-4e79-969c-31985090971a","Type":"ContainerStarted","Data":"62f9e32c3e745e8e16febd819c7f69acdb4e125640c40043b3eebb4e669a3dba"}
May 25 13:53:33.240385 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:33.240100    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq" podUID="915f615d-8e11-4584-bb10-57733fba7e83" containerName="switch-graph-12313" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:53:34.021117 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:34.021070    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc" podUID="a7e53380-9d1c-42f5-bbc3-4266185be4a0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
May 25 13:53:34.241099 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:34.241070    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"
May 25 13:53:34.413382 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:34.413342    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7e53380-9d1c-42f5-bbc3-4266185be4a0-proxy-tls\") pod \"a7e53380-9d1c-42f5-bbc3-4266185be4a0\" (UID: \"a7e53380-9d1c-42f5-bbc3-4266185be4a0\") "
May 25 13:53:34.413591 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:34.413394    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hg5g5\" (UniqueName: \"kubernetes.io/projected/a7e53380-9d1c-42f5-bbc3-4266185be4a0-kube-api-access-hg5g5\") pod \"a7e53380-9d1c-42f5-bbc3-4266185be4a0\" (UID: \"a7e53380-9d1c-42f5-bbc3-4266185be4a0\") "
May 25 13:53:34.413591 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:34.413423    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-12313-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7e53380-9d1c-42f5-bbc3-4266185be4a0-error-404-isvc-12313-kube-rbac-proxy-sar-config\") pod \"a7e53380-9d1c-42f5-bbc3-4266185be4a0\" (UID: \"a7e53380-9d1c-42f5-bbc3-4266185be4a0\") "
May 25 13:53:34.413895 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:34.413860    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e53380-9d1c-42f5-bbc3-4266185be4a0-error-404-isvc-12313-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-12313-kube-rbac-proxy-sar-config") pod "a7e53380-9d1c-42f5-bbc3-4266185be4a0" (UID: "a7e53380-9d1c-42f5-bbc3-4266185be4a0"). InnerVolumeSpecName "error-404-isvc-12313-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:53:34.415836 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:34.415779    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e53380-9d1c-42f5-bbc3-4266185be4a0-kube-api-access-hg5g5" (OuterVolumeSpecName: "kube-api-access-hg5g5") pod "a7e53380-9d1c-42f5-bbc3-4266185be4a0" (UID: "a7e53380-9d1c-42f5-bbc3-4266185be4a0"). InnerVolumeSpecName "kube-api-access-hg5g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:53:34.415968 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:34.415865    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7e53380-9d1c-42f5-bbc3-4266185be4a0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a7e53380-9d1c-42f5-bbc3-4266185be4a0" (UID: "a7e53380-9d1c-42f5-bbc3-4266185be4a0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:53:34.460536 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:34.460492    2569 generic.go:358] "Generic (PLEG): container finished" podID="a7e53380-9d1c-42f5-bbc3-4266185be4a0" containerID="65eed7ee6ce35f4bd51ba461b1bc93f48cefc31b8a179ec35dd835d249b8d35c" exitCode=0
May 25 13:53:34.460704 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:34.460580    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"
May 25 13:53:34.460704 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:34.460576    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc" event={"ID":"a7e53380-9d1c-42f5-bbc3-4266185be4a0","Type":"ContainerDied","Data":"65eed7ee6ce35f4bd51ba461b1bc93f48cefc31b8a179ec35dd835d249b8d35c"}
May 25 13:53:34.460704 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:34.460622    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc" event={"ID":"a7e53380-9d1c-42f5-bbc3-4266185be4a0","Type":"ContainerDied","Data":"6d6c7db41ff6f2194f1daf45e3aa94ad7b949a8b10716581d0e14438b10353ce"}
May 25 13:53:34.460704 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:34.460645    2569 scope.go:117] "RemoveContainer" containerID="6e9d5cc18fbe7d13369784000fe9f8d97c3be253e48424a9990702a3204b1c9b"
May 25 13:53:34.483659 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:34.483631    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"]
May 25 13:53:34.489443 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:34.489411    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12313-predictor-7746bc869-vlbsc"]
May 25 13:53:34.514562 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:34.514526    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7e53380-9d1c-42f5-bbc3-4266185be4a0-proxy-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:53:34.514562 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:34.514556    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg5g5\" (UniqueName: \"kubernetes.io/projected/a7e53380-9d1c-42f5-bbc3-4266185be4a0-kube-api-access-hg5g5\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:53:34.514562 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:34.514570    2569 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-12313-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7e53380-9d1c-42f5-bbc3-4266185be4a0-error-404-isvc-12313-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:53:34.597762 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:34.597730    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7e53380-9d1c-42f5-bbc3-4266185be4a0" path="/var/lib/kubelet/pods/a7e53380-9d1c-42f5-bbc3-4266185be4a0/volumes"
May 25 13:53:37.188279 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:37.188035    2569 scope.go:117] "RemoveContainer" containerID="65eed7ee6ce35f4bd51ba461b1bc93f48cefc31b8a179ec35dd835d249b8d35c"
May 25 13:53:38.239106 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:38.239065    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq" podUID="915f615d-8e11-4584-bb10-57733fba7e83" containerName="switch-graph-12313" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:53:38.280606 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:38.280416    2569 scope.go:117] "RemoveContainer" containerID="6e9d5cc18fbe7d13369784000fe9f8d97c3be253e48424a9990702a3204b1c9b"
May 25 13:53:38.280739 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:53:38.280719    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e9d5cc18fbe7d13369784000fe9f8d97c3be253e48424a9990702a3204b1c9b\": container with ID starting with 6e9d5cc18fbe7d13369784000fe9f8d97c3be253e48424a9990702a3204b1c9b not found: ID does not exist" containerID="6e9d5cc18fbe7d13369784000fe9f8d97c3be253e48424a9990702a3204b1c9b"
May 25 13:53:38.280789 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:38.280752    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e9d5cc18fbe7d13369784000fe9f8d97c3be253e48424a9990702a3204b1c9b"} err="failed to get container status \"6e9d5cc18fbe7d13369784000fe9f8d97c3be253e48424a9990702a3204b1c9b\": rpc error: code = NotFound desc = could not find container \"6e9d5cc18fbe7d13369784000fe9f8d97c3be253e48424a9990702a3204b1c9b\": container with ID starting with 6e9d5cc18fbe7d13369784000fe9f8d97c3be253e48424a9990702a3204b1c9b not found: ID does not exist"
May 25 13:53:38.280789 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:38.280775    2569 scope.go:117] "RemoveContainer" containerID="65eed7ee6ce35f4bd51ba461b1bc93f48cefc31b8a179ec35dd835d249b8d35c"
May 25 13:53:38.281069 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:53:38.281048    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65eed7ee6ce35f4bd51ba461b1bc93f48cefc31b8a179ec35dd835d249b8d35c\": container with ID starting with 65eed7ee6ce35f4bd51ba461b1bc93f48cefc31b8a179ec35dd835d249b8d35c not found: ID does not exist" containerID="65eed7ee6ce35f4bd51ba461b1bc93f48cefc31b8a179ec35dd835d249b8d35c"
May 25 13:53:38.281150 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:38.281072    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65eed7ee6ce35f4bd51ba461b1bc93f48cefc31b8a179ec35dd835d249b8d35c"} err="failed to get container status \"65eed7ee6ce35f4bd51ba461b1bc93f48cefc31b8a179ec35dd835d249b8d35c\": rpc error: code = NotFound desc = could not find container \"65eed7ee6ce35f4bd51ba461b1bc93f48cefc31b8a179ec35dd835d249b8d35c\": container with ID starting with 65eed7ee6ce35f4bd51ba461b1bc93f48cefc31b8a179ec35dd835d249b8d35c not found: ID does not exist"
May 25 13:53:39.486643 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:39.486601    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bg2bw/must-gather-qgxtt" event={"ID":"f722ffe9-8cfd-4e79-969c-31985090971a","Type":"ContainerStarted","Data":"5eae517426480dee87cf6f5ade2fe72f74812774aeaac37a3f5f16b63cee0b22"}
May 25 13:53:39.486643 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:39.486645    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bg2bw/must-gather-qgxtt" event={"ID":"f722ffe9-8cfd-4e79-969c-31985090971a","Type":"ContainerStarted","Data":"0600b789f2fb1389dbf209648123526aec83c8bf35e4064200008a159e501da5"}
May 25 13:53:39.505884 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:39.505837    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-bg2bw/must-gather-qgxtt" podStartSLOduration=2.31500425 podStartE2EDuration="8.505821809s" podCreationTimestamp="2026-05-25 13:53:31 +0000 UTC" firstStartedPulling="2026-05-25 13:53:32.206157374 +0000 UTC m=+2928.208899105" lastFinishedPulling="2026-05-25 13:53:38.396974918 +0000 UTC m=+2934.399716664" observedRunningTime="2026-05-25 13:53:39.504658386 +0000 UTC m=+2935.507400151" watchObservedRunningTime="2026-05-25 13:53:39.505821809 +0000 UTC m=+2935.508563631"
May 25 13:53:43.237599 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:43.237559    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq" podUID="915f615d-8e11-4584-bb10-57733fba7e83" containerName="switch-graph-12313" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:53:43.237983 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:43.237666    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq"
May 25 13:53:47.523825 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:47.523789    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-12313-76b55b9df6-96qlq_915f615d-8e11-4584-bb10-57733fba7e83/switch-graph-12313/0.log"
May 25 13:53:48.239215 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:48.239152    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq" podUID="915f615d-8e11-4584-bb10-57733fba7e83" containerName="switch-graph-12313" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:53:48.336434 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:48.336403    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-12313-76b55b9df6-96qlq_915f615d-8e11-4584-bb10-57733fba7e83/switch-graph-12313/0.log"
May 25 13:53:49.143549 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:49.143517    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-12313-76b55b9df6-96qlq_915f615d-8e11-4584-bb10-57733fba7e83/switch-graph-12313/0.log"
May 25 13:53:49.938991 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:49.938955    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-12313-76b55b9df6-96qlq_915f615d-8e11-4584-bb10-57733fba7e83/switch-graph-12313/0.log"
May 25 13:53:50.730338 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:50.730306    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-12313-76b55b9df6-96qlq_915f615d-8e11-4584-bb10-57733fba7e83/switch-graph-12313/0.log"
May 25 13:53:51.520433 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:51.520399    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-12313-76b55b9df6-96qlq_915f615d-8e11-4584-bb10-57733fba7e83/switch-graph-12313/0.log"
May 25 13:53:52.345924 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:52.345896    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-12313-76b55b9df6-96qlq_915f615d-8e11-4584-bb10-57733fba7e83/switch-graph-12313/0.log"
May 25 13:53:53.156628 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:53.156598    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-12313-76b55b9df6-96qlq_915f615d-8e11-4584-bb10-57733fba7e83/switch-graph-12313/0.log"
May 25 13:53:53.238737 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:53.238688    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq" podUID="915f615d-8e11-4584-bb10-57733fba7e83" containerName="switch-graph-12313" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:53:53.941777 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:53.941740    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-12313-76b55b9df6-96qlq_915f615d-8e11-4584-bb10-57733fba7e83/switch-graph-12313/0.log"
May 25 13:53:54.729397 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:54.729343    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-12313-76b55b9df6-96qlq_915f615d-8e11-4584-bb10-57733fba7e83/switch-graph-12313/0.log"
May 25 13:53:55.511235 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:55.511190    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-12313-76b55b9df6-96qlq_915f615d-8e11-4584-bb10-57733fba7e83/switch-graph-12313/0.log"
May 25 13:53:56.289367 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:56.289339    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-12313-76b55b9df6-96qlq_915f615d-8e11-4584-bb10-57733fba7e83/switch-graph-12313/0.log"
May 25 13:53:58.238081 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:53:58.238045    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq" podUID="915f615d-8e11-4584-bb10-57733fba7e83" containerName="switch-graph-12313" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:54:00.560116 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:00.560074    2569 generic.go:358] "Generic (PLEG): container finished" podID="f722ffe9-8cfd-4e79-969c-31985090971a" containerID="0600b789f2fb1389dbf209648123526aec83c8bf35e4064200008a159e501da5" exitCode=0
May 25 13:54:00.560534 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:00.560152    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bg2bw/must-gather-qgxtt" event={"ID":"f722ffe9-8cfd-4e79-969c-31985090971a","Type":"ContainerDied","Data":"0600b789f2fb1389dbf209648123526aec83c8bf35e4064200008a159e501da5"}
May 25 13:54:00.560534 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:00.560509    2569 scope.go:117] "RemoveContainer" containerID="0600b789f2fb1389dbf209648123526aec83c8bf35e4064200008a159e501da5"
May 25 13:54:00.727969 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:00.727945    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq"
May 25 13:54:00.836868 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:00.836829    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/915f615d-8e11-4584-bb10-57733fba7e83-openshift-service-ca-bundle\") pod \"915f615d-8e11-4584-bb10-57733fba7e83\" (UID: \"915f615d-8e11-4584-bb10-57733fba7e83\") "
May 25 13:54:00.837027 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:00.836892    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/915f615d-8e11-4584-bb10-57733fba7e83-proxy-tls\") pod \"915f615d-8e11-4584-bb10-57733fba7e83\" (UID: \"915f615d-8e11-4584-bb10-57733fba7e83\") "
May 25 13:54:00.837278 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:00.837258    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/915f615d-8e11-4584-bb10-57733fba7e83-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "915f615d-8e11-4584-bb10-57733fba7e83" (UID: "915f615d-8e11-4584-bb10-57733fba7e83"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:54:00.838963 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:00.838940    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/915f615d-8e11-4584-bb10-57733fba7e83-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "915f615d-8e11-4584-bb10-57733fba7e83" (UID: "915f615d-8e11-4584-bb10-57733fba7e83"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:54:00.937815 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:00.937722    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/915f615d-8e11-4584-bb10-57733fba7e83-openshift-service-ca-bundle\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:54:00.937815 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:00.937754    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/915f615d-8e11-4584-bb10-57733fba7e83-proxy-tls\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:54:01.404107 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:01.404076    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bg2bw_must-gather-qgxtt_f722ffe9-8cfd-4e79-969c-31985090971a/gather/0.log"
May 25 13:54:01.564059 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:01.564021    2569 generic.go:358] "Generic (PLEG): container finished" podID="915f615d-8e11-4584-bb10-57733fba7e83" containerID="a964ecb83ff75ae2870ed3bfa129e0755b96eb9799456c2b25f6667c45da2f37" exitCode=0
May 25 13:54:01.564534 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:01.564091    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq"
May 25 13:54:01.564534 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:01.564109    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq" event={"ID":"915f615d-8e11-4584-bb10-57733fba7e83","Type":"ContainerDied","Data":"a964ecb83ff75ae2870ed3bfa129e0755b96eb9799456c2b25f6667c45da2f37"}
May 25 13:54:01.564534 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:01.564154    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq" event={"ID":"915f615d-8e11-4584-bb10-57733fba7e83","Type":"ContainerDied","Data":"6a6abadd33c342807ef2f869eac8bffd9a7cd44dd7de574c0f8969ddc63756ee"}
May 25 13:54:01.564534 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:01.564184    2569 scope.go:117] "RemoveContainer" containerID="a964ecb83ff75ae2870ed3bfa129e0755b96eb9799456c2b25f6667c45da2f37"
May 25 13:54:01.573059 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:01.573038    2569 scope.go:117] "RemoveContainer" containerID="a964ecb83ff75ae2870ed3bfa129e0755b96eb9799456c2b25f6667c45da2f37"
May 25 13:54:01.573352 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:54:01.573331    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a964ecb83ff75ae2870ed3bfa129e0755b96eb9799456c2b25f6667c45da2f37\": container with ID starting with a964ecb83ff75ae2870ed3bfa129e0755b96eb9799456c2b25f6667c45da2f37 not found: ID does not exist" containerID="a964ecb83ff75ae2870ed3bfa129e0755b96eb9799456c2b25f6667c45da2f37"
May 25 13:54:01.573417 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:01.573365    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a964ecb83ff75ae2870ed3bfa129e0755b96eb9799456c2b25f6667c45da2f37"} err="failed to get container status \"a964ecb83ff75ae2870ed3bfa129e0755b96eb9799456c2b25f6667c45da2f37\": rpc error: code = NotFound desc = could not find container \"a964ecb83ff75ae2870ed3bfa129e0755b96eb9799456c2b25f6667c45da2f37\": container with ID starting with a964ecb83ff75ae2870ed3bfa129e0755b96eb9799456c2b25f6667c45da2f37 not found: ID does not exist"
May 25 13:54:01.585776 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:01.585752    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq"]
May 25 13:54:01.587121 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:01.587104    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-12313-76b55b9df6-96qlq"]
May 25 13:54:02.027115 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.027079    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q2qb9/must-gather-pg8dh"]
May 25 13:54:02.027449 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.027435    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7e53380-9d1c-42f5-bbc3-4266185be4a0" containerName="kserve-container"
May 25 13:54:02.027517 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.027449    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7e53380-9d1c-42f5-bbc3-4266185be4a0" containerName="kserve-container"
May 25 13:54:02.027517 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.027464    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7e53380-9d1c-42f5-bbc3-4266185be4a0" containerName="kube-rbac-proxy"
May 25 13:54:02.027517 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.027470    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7e53380-9d1c-42f5-bbc3-4266185be4a0" containerName="kube-rbac-proxy"
May 25 13:54:02.027517 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.027481    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="915f615d-8e11-4584-bb10-57733fba7e83" containerName="switch-graph-12313"
May 25 13:54:02.027517 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.027486    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="915f615d-8e11-4584-bb10-57733fba7e83" containerName="switch-graph-12313"
May 25 13:54:02.027678 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.027547    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="a7e53380-9d1c-42f5-bbc3-4266185be4a0" containerName="kube-rbac-proxy"
May 25 13:54:02.027678 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.027555    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="915f615d-8e11-4584-bb10-57733fba7e83" containerName="switch-graph-12313"
May 25 13:54:02.027678 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.027563    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="a7e53380-9d1c-42f5-bbc3-4266185be4a0" containerName="kserve-container"
May 25 13:54:02.031712 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.031691    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q2qb9/must-gather-pg8dh"
May 25 13:54:02.034051 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.034029    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q2qb9\"/\"default-dockercfg-pvqsp\""
May 25 13:54:02.034176 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.034061    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q2qb9\"/\"openshift-service-ca.crt\""
May 25 13:54:02.034311 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.034296    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q2qb9\"/\"kube-root-ca.crt\""
May 25 13:54:02.038462 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.038439    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q2qb9/must-gather-pg8dh"]
May 25 13:54:02.149335 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.149298    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b6c87135-5bbb-46f3-975d-62eb82fd097d-must-gather-output\") pod \"must-gather-pg8dh\" (UID: \"b6c87135-5bbb-46f3-975d-62eb82fd097d\") " pod="openshift-must-gather-q2qb9/must-gather-pg8dh"
May 25 13:54:02.149546 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.149365    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nbsp\" (UniqueName: \"kubernetes.io/projected/b6c87135-5bbb-46f3-975d-62eb82fd097d-kube-api-access-7nbsp\") pod \"must-gather-pg8dh\" (UID: \"b6c87135-5bbb-46f3-975d-62eb82fd097d\") " pod="openshift-must-gather-q2qb9/must-gather-pg8dh"
May 25 13:54:02.250448 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.250409    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b6c87135-5bbb-46f3-975d-62eb82fd097d-must-gather-output\") pod \"must-gather-pg8dh\" (UID: \"b6c87135-5bbb-46f3-975d-62eb82fd097d\") " pod="openshift-must-gather-q2qb9/must-gather-pg8dh"
May 25 13:54:02.250607 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.250463    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7nbsp\" (UniqueName: \"kubernetes.io/projected/b6c87135-5bbb-46f3-975d-62eb82fd097d-kube-api-access-7nbsp\") pod \"must-gather-pg8dh\" (UID: \"b6c87135-5bbb-46f3-975d-62eb82fd097d\") " pod="openshift-must-gather-q2qb9/must-gather-pg8dh"
May 25 13:54:02.250812 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.250790    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b6c87135-5bbb-46f3-975d-62eb82fd097d-must-gather-output\") pod \"must-gather-pg8dh\" (UID: \"b6c87135-5bbb-46f3-975d-62eb82fd097d\") " pod="openshift-must-gather-q2qb9/must-gather-pg8dh"
May 25 13:54:02.258365 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.258331    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nbsp\" (UniqueName: \"kubernetes.io/projected/b6c87135-5bbb-46f3-975d-62eb82fd097d-kube-api-access-7nbsp\") pod \"must-gather-pg8dh\" (UID: \"b6c87135-5bbb-46f3-975d-62eb82fd097d\") " pod="openshift-must-gather-q2qb9/must-gather-pg8dh"
May 25 13:54:02.341501 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.341393    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q2qb9/must-gather-pg8dh"
May 25 13:54:02.470377 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.470342    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q2qb9/must-gather-pg8dh"]
May 25 13:54:02.476010 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:54:02.475975    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6c87135_5bbb_46f3_975d_62eb82fd097d.slice/crio-6a6ca573a9e88ac3511bc7124aaa0d0060d3f46790fa284679fc5abe0de367fb WatchSource:0}: Error finding container 6a6ca573a9e88ac3511bc7124aaa0d0060d3f46790fa284679fc5abe0de367fb: Status 404 returned error can't find the container with id 6a6ca573a9e88ac3511bc7124aaa0d0060d3f46790fa284679fc5abe0de367fb
May 25 13:54:02.567740 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.567705    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q2qb9/must-gather-pg8dh" event={"ID":"b6c87135-5bbb-46f3-975d-62eb82fd097d","Type":"ContainerStarted","Data":"6a6ca573a9e88ac3511bc7124aaa0d0060d3f46790fa284679fc5abe0de367fb"}
May 25 13:54:02.597489 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:02.597456    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="915f615d-8e11-4584-bb10-57733fba7e83" path="/var/lib/kubelet/pods/915f615d-8e11-4584-bb10-57733fba7e83/volumes"
May 25 13:54:03.575256 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:03.575130    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q2qb9/must-gather-pg8dh" event={"ID":"b6c87135-5bbb-46f3-975d-62eb82fd097d","Type":"ContainerStarted","Data":"b433efe25938846162266884484624c2979ca0543c8ab65bb5b5414393fa0878"}
May 25 13:54:03.575256 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:03.575201    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q2qb9/must-gather-pg8dh" event={"ID":"b6c87135-5bbb-46f3-975d-62eb82fd097d","Type":"ContainerStarted","Data":"f837407d1c137787c5f278c4796d44f246a37751398951735727a3e4e867e548"}
May 25 13:54:03.591453 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:03.591337    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-q2qb9/must-gather-pg8dh" podStartSLOduration=0.810190337 podStartE2EDuration="1.591316352s" podCreationTimestamp="2026-05-25 13:54:02 +0000 UTC" firstStartedPulling="2026-05-25 13:54:02.477846675 +0000 UTC m=+2958.480588407" lastFinishedPulling="2026-05-25 13:54:03.258972685 +0000 UTC m=+2959.261714422" observedRunningTime="2026-05-25 13:54:03.590458293 +0000 UTC m=+2959.593200058" watchObservedRunningTime="2026-05-25 13:54:03.591316352 +0000 UTC m=+2959.594058108"
May 25 13:54:04.621088 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:04.621054    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-27qqj_e1da9cca-c41c-4afc-955f-46df4c30292a/global-pull-secret-syncer/0.log"
May 25 13:54:04.837018 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:04.836966    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vsr8s_e69ac0d2-7a7a-496b-ba07-c08a6507fd39/konnectivity-agent/0.log"
May 25 13:54:04.903642 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:04.903561    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-18.ec2.internal_e6365ea56860ef681cb6ea98ee36074a/haproxy/0.log"
May 25 13:54:06.849927 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:06.849856    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-bg2bw/must-gather-qgxtt"]
May 25 13:54:06.850597 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:06.850300    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-bg2bw/must-gather-qgxtt" podUID="f722ffe9-8cfd-4e79-969c-31985090971a" containerName="copy" containerID="cri-o://5eae517426480dee87cf6f5ade2fe72f74812774aeaac37a3f5f16b63cee0b22" gracePeriod=2
May 25 13:54:06.852716 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:06.852675    2569 status_manager.go:895] "Failed to get status for pod" podUID="f722ffe9-8cfd-4e79-969c-31985090971a" pod="openshift-must-gather-bg2bw/must-gather-qgxtt" err="pods \"must-gather-qgxtt\" is forbidden: User \"system:node:ip-10-0-139-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bg2bw\": no relationship found between node 'ip-10-0-139-18.ec2.internal' and this object"
May 25 13:54:06.852965 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:06.852798    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-bg2bw/must-gather-qgxtt"]
May 25 13:54:07.242711 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:07.242677    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bg2bw_must-gather-qgxtt_f722ffe9-8cfd-4e79-969c-31985090971a/copy/0.log"
May 25 13:54:07.243117 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:07.243096    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bg2bw/must-gather-qgxtt"
May 25 13:54:07.245315 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:07.245275    2569 status_manager.go:895] "Failed to get status for pod" podUID="f722ffe9-8cfd-4e79-969c-31985090971a" pod="openshift-must-gather-bg2bw/must-gather-qgxtt" err="pods \"must-gather-qgxtt\" is forbidden: User \"system:node:ip-10-0-139-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bg2bw\": no relationship found between node 'ip-10-0-139-18.ec2.internal' and this object"
May 25 13:54:07.296709 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:07.296669    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpcnz\" (UniqueName: \"kubernetes.io/projected/f722ffe9-8cfd-4e79-969c-31985090971a-kube-api-access-qpcnz\") pod \"f722ffe9-8cfd-4e79-969c-31985090971a\" (UID: \"f722ffe9-8cfd-4e79-969c-31985090971a\") "
May 25 13:54:07.297066 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:07.297010    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f722ffe9-8cfd-4e79-969c-31985090971a-must-gather-output\") pod \"f722ffe9-8cfd-4e79-969c-31985090971a\" (UID: \"f722ffe9-8cfd-4e79-969c-31985090971a\") "
May 25 13:54:07.300340 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:07.300307    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f722ffe9-8cfd-4e79-969c-31985090971a-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "f722ffe9-8cfd-4e79-969c-31985090971a" (UID: "f722ffe9-8cfd-4e79-969c-31985090971a"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:54:07.304341 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:07.304280    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f722ffe9-8cfd-4e79-969c-31985090971a-kube-api-access-qpcnz" (OuterVolumeSpecName: "kube-api-access-qpcnz") pod "f722ffe9-8cfd-4e79-969c-31985090971a" (UID: "f722ffe9-8cfd-4e79-969c-31985090971a"). InnerVolumeSpecName "kube-api-access-qpcnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:54:07.397974 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:07.397864    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpcnz\" (UniqueName: \"kubernetes.io/projected/f722ffe9-8cfd-4e79-969c-31985090971a-kube-api-access-qpcnz\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:54:07.397974 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:07.397913    2569 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f722ffe9-8cfd-4e79-969c-31985090971a-must-gather-output\") on node \"ip-10-0-139-18.ec2.internal\" DevicePath \"\""
May 25 13:54:07.591717 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:07.591681    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bg2bw_must-gather-qgxtt_f722ffe9-8cfd-4e79-969c-31985090971a/copy/0.log"
May 25 13:54:07.592118 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:07.592092    2569 generic.go:358] "Generic (PLEG): container finished" podID="f722ffe9-8cfd-4e79-969c-31985090971a" containerID="5eae517426480dee87cf6f5ade2fe72f74812774aeaac37a3f5f16b63cee0b22" exitCode=143
May 25 13:54:07.592290 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:07.592274    2569 scope.go:117] "RemoveContainer" containerID="5eae517426480dee87cf6f5ade2fe72f74812774aeaac37a3f5f16b63cee0b22"
May 25 13:54:07.592464 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:07.592448    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bg2bw/must-gather-qgxtt"
May 25 13:54:07.601474 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:07.601423    2569 status_manager.go:895] "Failed to get status for pod" podUID="f722ffe9-8cfd-4e79-969c-31985090971a" pod="openshift-must-gather-bg2bw/must-gather-qgxtt" err="pods \"must-gather-qgxtt\" is forbidden: User \"system:node:ip-10-0-139-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bg2bw\": no relationship found between node 'ip-10-0-139-18.ec2.internal' and this object"
May 25 13:54:07.614419 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:07.614369    2569 status_manager.go:895] "Failed to get status for pod" podUID="f722ffe9-8cfd-4e79-969c-31985090971a" pod="openshift-must-gather-bg2bw/must-gather-qgxtt" err="pods \"must-gather-qgxtt\" is forbidden: User \"system:node:ip-10-0-139-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bg2bw\": no relationship found between node 'ip-10-0-139-18.ec2.internal' and this object"
May 25 13:54:07.615405 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:07.615041    2569 scope.go:117] "RemoveContainer" containerID="0600b789f2fb1389dbf209648123526aec83c8bf35e4064200008a159e501da5"
May 25 13:54:07.635838 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:07.635550    2569 scope.go:117] "RemoveContainer" containerID="5eae517426480dee87cf6f5ade2fe72f74812774aeaac37a3f5f16b63cee0b22"
May 25 13:54:07.636416 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:54:07.636090    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eae517426480dee87cf6f5ade2fe72f74812774aeaac37a3f5f16b63cee0b22\": container with ID starting with 5eae517426480dee87cf6f5ade2fe72f74812774aeaac37a3f5f16b63cee0b22 not found: ID does not exist" containerID="5eae517426480dee87cf6f5ade2fe72f74812774aeaac37a3f5f16b63cee0b22"
May 25 13:54:07.636416 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:07.636129    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eae517426480dee87cf6f5ade2fe72f74812774aeaac37a3f5f16b63cee0b22"} err="failed to get container status \"5eae517426480dee87cf6f5ade2fe72f74812774aeaac37a3f5f16b63cee0b22\": rpc error: code = NotFound desc = could not find container \"5eae517426480dee87cf6f5ade2fe72f74812774aeaac37a3f5f16b63cee0b22\": container with ID starting with 5eae517426480dee87cf6f5ade2fe72f74812774aeaac37a3f5f16b63cee0b22 not found: ID does not exist"
May 25 13:54:07.636416 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:07.636154    2569 scope.go:117] "RemoveContainer" containerID="0600b789f2fb1389dbf209648123526aec83c8bf35e4064200008a159e501da5"
May 25 13:54:07.636815 ip-10-0-139-18 kubenswrapper[2569]: E0525 13:54:07.636724    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0600b789f2fb1389dbf209648123526aec83c8bf35e4064200008a159e501da5\": container with ID starting with 0600b789f2fb1389dbf209648123526aec83c8bf35e4064200008a159e501da5 not found: ID does not exist" containerID="0600b789f2fb1389dbf209648123526aec83c8bf35e4064200008a159e501da5"
May 25 13:54:07.636815 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:07.636753    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0600b789f2fb1389dbf209648123526aec83c8bf35e4064200008a159e501da5"} err="failed to get container status \"0600b789f2fb1389dbf209648123526aec83c8bf35e4064200008a159e501da5\": rpc error: code = NotFound desc = could not find container \"0600b789f2fb1389dbf209648123526aec83c8bf35e4064200008a159e501da5\": container with ID starting with 0600b789f2fb1389dbf209648123526aec83c8bf35e4064200008a159e501da5 not found: ID does not exist"
May 25 13:54:08.599384 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:08.598842    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f722ffe9-8cfd-4e79-969c-31985090971a" path="/var/lib/kubelet/pods/f722ffe9-8cfd-4e79-969c-31985090971a/volumes"
May 25 13:54:08.599925 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:08.599387    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6c75959455-hb7qq_3d09042c-c990-4d33-8e6c-9aba7f5a94d7/cluster-monitoring-operator/0.log"
May 25 13:54:08.631372 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:08.631339    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-znb4d_ff44f516-493d-4bab-8b9c-841c79d799a7/kube-state-metrics/0.log"
May 25 13:54:08.658666 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:08.658624    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-znb4d_ff44f516-493d-4bab-8b9c-841c79d799a7/kube-rbac-proxy-main/0.log"
May 25 13:54:08.686805 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:08.686777    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-znb4d_ff44f516-493d-4bab-8b9c-841c79d799a7/kube-rbac-proxy-self/0.log"
May 25 13:54:08.946160 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:08.946058    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x7b5z_f1152b71-6f95-4a41-87e0-98fccf6c5512/node-exporter/0.log"
May 25 13:54:08.968764 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:08.968716    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x7b5z_f1152b71-6f95-4a41-87e0-98fccf6c5512/kube-rbac-proxy/0.log"
May 25 13:54:08.990928 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:08.990836    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x7b5z_f1152b71-6f95-4a41-87e0-98fccf6c5512/init-textfile/0.log"
May 25 13:54:09.019654 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:09.019621    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-8w47r_b8e9b997-2768-47e6-82fb-5f498706590e/kube-rbac-proxy-main/0.log"
May 25 13:54:09.050971 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:09.050939    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-8w47r_b8e9b997-2768-47e6-82fb-5f498706590e/kube-rbac-proxy-self/0.log"
May 25 13:54:09.079755 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:09.079720    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-8w47r_b8e9b997-2768-47e6-82fb-5f498706590e/openshift-state-metrics/0.log"
May 25 13:54:09.125267 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:09.125234    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c7d8f3fe-873f-46ff-a897-4912c1434fd7/prometheus/0.log"
May 25 13:54:09.150995 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:09.150962    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c7d8f3fe-873f-46ff-a897-4912c1434fd7/config-reloader/0.log"
May 25 13:54:09.194793 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:09.194766    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c7d8f3fe-873f-46ff-a897-4912c1434fd7/thanos-sidecar/0.log"
May 25 13:54:09.222609 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:09.222508    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c7d8f3fe-873f-46ff-a897-4912c1434fd7/kube-rbac-proxy-web/0.log"
May 25 13:54:09.246519 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:09.246491    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c7d8f3fe-873f-46ff-a897-4912c1434fd7/kube-rbac-proxy/0.log"
May 25 13:54:09.270355 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:09.270318    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c7d8f3fe-873f-46ff-a897-4912c1434fd7/kube-rbac-proxy-thanos/0.log"
May 25 13:54:09.296943 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:09.296908    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c7d8f3fe-873f-46ff-a897-4912c1434fd7/init-config-reloader/0.log"
May 25 13:54:09.385920 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:09.385877    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-dzgcp_ad52974f-66ee-4440-b0ca-065f6d759892/prometheus-operator-admission-webhook/0.log"
May 25 13:54:09.505855 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:09.505771    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75d748b86c-j78qn_5109856c-fa52-4fc8-855d-ca1dff292b04/thanos-query/0.log"
May 25 13:54:09.531675 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:09.531640    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75d748b86c-j78qn_5109856c-fa52-4fc8-855d-ca1dff292b04/kube-rbac-proxy-web/0.log"
May 25 13:54:09.562496 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:09.562462    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75d748b86c-j78qn_5109856c-fa52-4fc8-855d-ca1dff292b04/kube-rbac-proxy/0.log"
May 25 13:54:09.594968 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:09.594933    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75d748b86c-j78qn_5109856c-fa52-4fc8-855d-ca1dff292b04/prom-label-proxy/0.log"
May 25 13:54:09.626956 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:09.626925    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75d748b86c-j78qn_5109856c-fa52-4fc8-855d-ca1dff292b04/kube-rbac-proxy-rules/0.log"
May 25 13:54:09.659778 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:09.659740    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75d748b86c-j78qn_5109856c-fa52-4fc8-855d-ca1dff292b04/kube-rbac-proxy-metrics/0.log"
May 25 13:54:11.209309 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:11.209248    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/1.log"
May 25 13:54:11.217447 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:11.217412    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-8wljq_feba83ac-e3f6-44e8-bd44-14aa150789a2/console-operator/2.log"
May 25 13:54:11.606691 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:11.606661    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-57bd96b888-gw2fw_b17a540d-e750-4fc7-b014-b5905162bad2/download-server/0.log"
May 25 13:54:11.998586 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:11.998512    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5cf8c89d64-dzlqc_62b6e938-f09a-4ed8-8e4a-a2135505f9a8/volume-data-source-validator/0.log"
May 25 13:54:12.106198 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.106141    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4"]
May 25 13:54:12.106555 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.106538    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f722ffe9-8cfd-4e79-969c-31985090971a" containerName="copy"
May 25 13:54:12.106555 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.106556    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f722ffe9-8cfd-4e79-969c-31985090971a" containerName="copy"
May 25 13:54:12.106678 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.106577    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f722ffe9-8cfd-4e79-969c-31985090971a" containerName="gather"
May 25 13:54:12.106678 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.106583    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f722ffe9-8cfd-4e79-969c-31985090971a" containerName="gather"
May 25 13:54:12.106678 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.106636    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="f722ffe9-8cfd-4e79-969c-31985090971a" containerName="copy"
May 25 13:54:12.106678 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.106647    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="f722ffe9-8cfd-4e79-969c-31985090971a" containerName="gather"
May 25 13:54:12.111021 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.110990    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4"
May 25 13:54:12.117606 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.117576    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4"]
May 25 13:54:12.243713 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.243646    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/df6c0d51-b524-4ba4-8541-6b041c9619f0-proc\") pod \"perf-node-gather-daemonset-kklg4\" (UID: \"df6c0d51-b524-4ba4-8541-6b041c9619f0\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4"
May 25 13:54:12.244236 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.243774    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/df6c0d51-b524-4ba4-8541-6b041c9619f0-podres\") pod \"perf-node-gather-daemonset-kklg4\" (UID: \"df6c0d51-b524-4ba4-8541-6b041c9619f0\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4"
May 25 13:54:12.244236 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.243871    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9rgk\" (UniqueName: \"kubernetes.io/projected/df6c0d51-b524-4ba4-8541-6b041c9619f0-kube-api-access-g9rgk\") pod \"perf-node-gather-daemonset-kklg4\" (UID: \"df6c0d51-b524-4ba4-8541-6b041c9619f0\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4"
May 25 13:54:12.244236 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.243938    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df6c0d51-b524-4ba4-8541-6b041c9619f0-sys\") pod \"perf-node-gather-daemonset-kklg4\" (UID: \"df6c0d51-b524-4ba4-8541-6b041c9619f0\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4"
May 25 13:54:12.244236 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.244002    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df6c0d51-b524-4ba4-8541-6b041c9619f0-lib-modules\") pod \"perf-node-gather-daemonset-kklg4\" (UID: \"df6c0d51-b524-4ba4-8541-6b041c9619f0\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4"
May 25 13:54:12.345592 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.345557    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/df6c0d51-b524-4ba4-8541-6b041c9619f0-proc\") pod \"perf-node-gather-daemonset-kklg4\" (UID: \"df6c0d51-b524-4ba4-8541-6b041c9619f0\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4"
May 25 13:54:12.345778 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.345603    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/df6c0d51-b524-4ba4-8541-6b041c9619f0-podres\") pod \"perf-node-gather-daemonset-kklg4\" (UID: \"df6c0d51-b524-4ba4-8541-6b041c9619f0\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4"
May 25 13:54:12.345778 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.345637    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g9rgk\" (UniqueName: \"kubernetes.io/projected/df6c0d51-b524-4ba4-8541-6b041c9619f0-kube-api-access-g9rgk\") pod \"perf-node-gather-daemonset-kklg4\" (UID: \"df6c0d51-b524-4ba4-8541-6b041c9619f0\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4"
May 25 13:54:12.345778 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.345671    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df6c0d51-b524-4ba4-8541-6b041c9619f0-sys\") pod \"perf-node-gather-daemonset-kklg4\" (UID: \"df6c0d51-b524-4ba4-8541-6b041c9619f0\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4"
May 25 13:54:12.345778 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.345683    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/df6c0d51-b524-4ba4-8541-6b041c9619f0-proc\") pod \"perf-node-gather-daemonset-kklg4\" (UID: \"df6c0d51-b524-4ba4-8541-6b041c9619f0\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4"
May 25 13:54:12.345778 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.345727    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df6c0d51-b524-4ba4-8541-6b041c9619f0-sys\") pod \"perf-node-gather-daemonset-kklg4\" (UID: \"df6c0d51-b524-4ba4-8541-6b041c9619f0\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4"
May 25 13:54:12.345778 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.345761    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df6c0d51-b524-4ba4-8541-6b041c9619f0-lib-modules\") pod \"perf-node-gather-daemonset-kklg4\" (UID: \"df6c0d51-b524-4ba4-8541-6b041c9619f0\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4"
May 25 13:54:12.346111 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.345782    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/df6c0d51-b524-4ba4-8541-6b041c9619f0-podres\") pod \"perf-node-gather-daemonset-kklg4\" (UID: \"df6c0d51-b524-4ba4-8541-6b041c9619f0\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4"
May 25 13:54:12.346111 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.345884    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df6c0d51-b524-4ba4-8541-6b041c9619f0-lib-modules\") pod \"perf-node-gather-daemonset-kklg4\" (UID: \"df6c0d51-b524-4ba4-8541-6b041c9619f0\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4"
May 25 13:54:12.353832 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.353801    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9rgk\" (UniqueName: \"kubernetes.io/projected/df6c0d51-b524-4ba4-8541-6b041c9619f0-kube-api-access-g9rgk\") pod \"perf-node-gather-daemonset-kklg4\" (UID: \"df6c0d51-b524-4ba4-8541-6b041c9619f0\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4"
May 25 13:54:12.423364 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.423327    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4"
May 25 13:54:12.558973 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.558941    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4"]
May 25 13:54:12.562456 ip-10-0-139-18 kubenswrapper[2569]: W0525 13:54:12.562427    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddf6c0d51_b524_4ba4_8541_6b041c9619f0.slice/crio-1220cb17a7c05ed0c04a3e6f308e61b4fb865c158bc6a4a054736ff7bfc64697 WatchSource:0}: Error finding container 1220cb17a7c05ed0c04a3e6f308e61b4fb865c158bc6a4a054736ff7bfc64697: Status 404 returned error can't find the container with id 1220cb17a7c05ed0c04a3e6f308e61b4fb865c158bc6a4a054736ff7bfc64697
May 25 13:54:12.638343 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.637946    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4" event={"ID":"df6c0d51-b524-4ba4-8541-6b041c9619f0","Type":"ContainerStarted","Data":"1220cb17a7c05ed0c04a3e6f308e61b4fb865c158bc6a4a054736ff7bfc64697"}
May 25 13:54:12.684133 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.684105    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-27ppb_d0b340d4-ffa3-4bf8-aeda-9d13cf88d293/dns/0.log"
May 25 13:54:12.702956 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.702923    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-27ppb_d0b340d4-ffa3-4bf8-aeda-9d13cf88d293/kube-rbac-proxy/0.log"
May 25 13:54:12.842312 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:12.842283    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pz7qk_217de8ce-783f-48b4-bee9-82bbe61f2450/dns-node-resolver/0.log"
May 25 13:54:13.292515 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:13.292481    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c2xxn_d4c54f53-586b-496f-bd70-663617ee0a12/node-ca/0.log"
May 25 13:54:13.645910 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:13.645869    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4" event={"ID":"df6c0d51-b524-4ba4-8541-6b041c9619f0","Type":"ContainerStarted","Data":"f3dafffd7de02e9c1775bfc01e0ad05ec430c797bb2a563eb528966650be4c36"}
May 25 13:54:13.646363 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:13.646331    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4"
May 25 13:54:13.663604 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:13.663539    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4" podStartSLOduration=1.663518748 podStartE2EDuration="1.663518748s" podCreationTimestamp="2026-05-25 13:54:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:54:13.662136104 +0000 UTC m=+2969.664877859" watchObservedRunningTime="2026-05-25 13:54:13.663518748 +0000 UTC m=+2969.666260503"
May 25 13:54:14.349792 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:14.349756    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7v2wh_8b9092df-01f8-4d59-82f8-a058e84b55f5/serve-healthcheck-canary/0.log"
May 25 13:54:14.764523 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:14.764442    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5qn7d_ce26be33-f950-446a-9800-9b471d4c8bfe/kube-rbac-proxy/0.log"
May 25 13:54:14.784339 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:14.784308    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5qn7d_ce26be33-f950-446a-9800-9b471d4c8bfe/exporter/0.log"
May 25 13:54:14.805358 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:14.805330    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5qn7d_ce26be33-f950-446a-9800-9b471d4c8bfe/extractor/0.log"
May 25 13:54:17.243907 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:17.243873    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-rgssw_33488dd0-5667-45b0-8fba-5375bb11f1c7/manager/0.log"
May 25 13:54:17.264325 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:17.264301    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-5pskx_dbab6580-d2ae-49e4-93c5-1d7c345f3e44/s3-init/0.log"
May 25 13:54:19.660194 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:19.660139    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-kklg4"
May 25 13:54:21.211875 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:21.211825    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-j448b_0ad976e4-ddc2-4b60-b40f-d20b5b8a6b96/migrator/0.log"
May 25 13:54:21.252392 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:21.252308    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-j448b_0ad976e4-ddc2-4b60-b40f-d20b5b8a6b96/graceful-termination/0.log"
May 25 13:54:21.615382 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:21.615350    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-7spmp_0d406e4f-9b85-403f-b250-242f6d50a43f/kube-storage-version-migrator-operator/1.log"
May 25 13:54:21.616459 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:21.616440    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-7spmp_0d406e4f-9b85-403f-b250-242f6d50a43f/kube-storage-version-migrator-operator/0.log"
May 25 13:54:23.083081 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:23.083052    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x6gsv_07666b34-380e-47af-be65-da7bd0bc3611/kube-multus-additional-cni-plugins/0.log"
May 25 13:54:23.121247 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:23.121216    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x6gsv_07666b34-380e-47af-be65-da7bd0bc3611/egress-router-binary-copy/0.log"
May 25 13:54:23.164338 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:23.164308    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x6gsv_07666b34-380e-47af-be65-da7bd0bc3611/cni-plugins/0.log"
May 25 13:54:23.202777 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:23.202752    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x6gsv_07666b34-380e-47af-be65-da7bd0bc3611/bond-cni-plugin/0.log"
May 25 13:54:23.241830 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:23.241800    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x6gsv_07666b34-380e-47af-be65-da7bd0bc3611/routeoverride-cni/0.log"
May 25 13:54:23.283054 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:23.282994    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x6gsv_07666b34-380e-47af-be65-da7bd0bc3611/whereabouts-cni-bincopy/0.log"
May 25 13:54:23.325533 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:23.325448    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x6gsv_07666b34-380e-47af-be65-da7bd0bc3611/whereabouts-cni/0.log"
May 25 13:54:23.386624 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:23.386591    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-h95n5_3375ebf6-cae9-4704-8051-68847e838a73/kube-multus/0.log"
May 25 13:54:23.483024 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:23.482987    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8rxxc_8878a1b9-7848-4f10-a0f6-f88059209e87/network-metrics-daemon/0.log"
May 25 13:54:23.503182 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:23.503144    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8rxxc_8878a1b9-7848-4f10-a0f6-f88059209e87/kube-rbac-proxy/0.log"
May 25 13:54:24.573258 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:24.573226    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-controller/0.log"
May 25 13:54:24.589231 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:24.589193    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/0.log"
May 25 13:54:24.604869 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:24.604841    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovn-acl-logging/1.log"
May 25 13:54:24.624436 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:24.624403    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/kube-rbac-proxy-node/0.log"
May 25 13:54:24.644942 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:24.644911    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/kube-rbac-proxy-ovn-metrics/0.log"
May 25 13:54:24.668536 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:24.668506    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/northd/0.log"
May 25 13:54:24.688425 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:24.688395    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/nbdb/0.log"
May 25 13:54:24.708577 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:24.708547    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/sbdb/0.log"
May 25 13:54:24.825723 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:24.825638    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kgc9_f37fe088-db90-44e2-9418-7be91eb347ca/ovnkube-controller/0.log"
May 25 13:54:26.217243 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:26.217212    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-5ppwf_6a51a8f7-c97c-4de0-b36d-6821814e02d8/network-check-target-container/0.log"
May 25 13:54:27.079657 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:27.079627    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-4lc6w_5c9c6100-4de5-4d5a-a25f-9b384efadd19/iptables-alerter/0.log"
May 25 13:54:27.737506 ip-10-0-139-18 kubenswrapper[2569]: I0525 13:54:27.737482    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-z848n_713ea544-1d7a-4451-8439-751bf93048d1/tuned/0.log"