May 20 13:47:58.199584 ip-10-0-136-75 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 20 13:47:58.199607 ip-10-0-136-75 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 20 13:47:58.199616 ip-10-0-136-75 systemd[1]: kubelet.service: Failed with result 'resources'.
May 20 13:47:58.199917 ip-10-0-136-75 systemd[1]: Failed to start Kubernetes Kubelet.
May 20 13:48:09.563886 ip-10-0-136-75 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 20 13:48:09.563909 ip-10-0-136-75 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6f077f17d3ac4fa29acc071ef3cbd57a --
May 20 13:50:32.626899 ip-10-0-136-75 systemd[1]: Starting Kubernetes Kubelet...
May 20 13:50:33.101498 ip-10-0-136-75 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 13:50:33.101498 ip-10-0-136-75 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 20 13:50:33.101498 ip-10-0-136-75 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 13:50:33.101498 ip-10-0-136-75 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 20 13:50:33.101498 ip-10-0-136-75 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 13:50:33.104643 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.104529    2570 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 20 13:50:33.107291 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107275    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:33.107331 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107293    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:33.107331 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107297    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:33.107331 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107302    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:33.107331 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107307    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:33.107331 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107311    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:33.107331 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107315    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:33.107331 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107319    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:33.107331 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107322    2570 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:33.107331 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107325    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:33.107331 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107327    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:33.107331 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107330    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:33.107331 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107333    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:33.107331 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107336    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:33.107725 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107339    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:33.107725 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107341    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:33.107725 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107344    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:33.107725 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107347    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:33.107725 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107349    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:33.107725 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107352    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:33.107725 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107355    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:33.107725 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107357    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:33.107725 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107360    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:33.107725 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107362    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:33.107725 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107365    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:33.107725 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107367    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:33.107725 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107369    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:33.107725 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107373    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:33.107725 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107375    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:33.107725 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107379    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:33.107725 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107382    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:33.107725 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107385    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:33.107725 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107388    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:33.107725 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107391    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:33.108203 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107394    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:33.108203 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107397    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:33.108203 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107399    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:33.108203 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107402    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:33.108203 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107404    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:33.108203 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107407    2570 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:33.108203 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107410    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:33.108203 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107412    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:33.108203 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107415    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:33.108203 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107417    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:33.108203 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107420    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:33.108203 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107422    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:33.108203 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107425    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:33.108203 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107429    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:33.108203 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107432    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:33.108203 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107434    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:33.108203 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107437    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:33.108203 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107439    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:33.108203 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107443    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:33.108203 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107447    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:33.108728 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107451    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:33.108728 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107454    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:33.108728 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107456    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:33.108728 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107459    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:33.108728 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107462    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:33.108728 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107465    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:33.108728 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107467    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:33.108728 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107470    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:33.108728 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107473    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:33.108728 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107475    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:33.108728 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107478    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:33.108728 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107480    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:33.108728 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107484    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:33.108728 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107486    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:33.108728 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107489    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:33.108728 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107491    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:33.108728 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107494    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:33.108728 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107496    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:33.108728 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107498    2570 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:33.109160 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107501    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:33.109160 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107503    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:33.109160 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107506    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:33.109160 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107508    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:33.109160 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107510    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:33.109160 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107513    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:33.109160 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107516    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:33.109160 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107519    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:33.109160 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107522    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:33.109160 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107524    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:33.109160 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107527    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:33.109160 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107530    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:33.109160 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107532    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:33.109160 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107975    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:33.109160 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107981    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:33.109160 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107984    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:33.109160 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107987    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:33.109160 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107990    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:33.109160 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107994    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:33.109160 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.107998    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:33.109671 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108001    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:33.109671 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108004    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:33.109671 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108007    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:33.109671 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108009    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:33.109671 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108012    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:33.109671 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108015    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:33.109671 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108018    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:33.109671 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108021    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:33.109671 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108024    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:33.109671 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108026    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:33.109671 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108029    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:33.109671 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108031    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:33.109671 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108034    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:33.109671 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108036    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:33.109671 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108039    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:33.109671 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108041    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:33.109671 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108044    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:33.109671 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108047    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:33.109671 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108051    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:33.109671 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108053    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:33.110175 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108056    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:33.110175 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108059    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:33.110175 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108061    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:33.110175 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108064    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:33.110175 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108067    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:33.110175 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108069    2570 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:33.110175 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108072    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:33.110175 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108074    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:33.110175 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108077    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:33.110175 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108079    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:33.110175 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108082    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:33.110175 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108084    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:33.110175 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108087    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:33.110175 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108090    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:33.110175 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108092    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:33.110175 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108095    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:33.110175 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108097    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:33.110175 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108100    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:33.110175 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108105    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:33.110175 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108108    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:33.110665 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108110    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:33.110665 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108113    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:33.110665 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108115    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:33.110665 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108118    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:33.110665 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108121    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:33.110665 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108123    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:33.110665 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108125    2570 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:33.110665 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108128    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:33.110665 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108130    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:33.110665 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108133    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:33.110665 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108139    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:33.110665 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108142    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:33.110665 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108146    2570 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:33.110665 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108149    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:33.110665 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108153    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:33.110665 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108155    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:33.110665 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108159    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:33.110665 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108162    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:33.110665 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108164    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:33.110665 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108167    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:33.111136 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108169    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:33.111136 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108172    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:33.111136 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108175    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:33.111136 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108177    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:33.111136 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108179    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:33.111136 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108182    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:33.111136 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108185    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:33.111136 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108187    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:33.111136 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108190    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:33.111136 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108193    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:33.111136 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108195    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:33.111136 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108198    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:33.111136 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108201    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:33.111136 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108203    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:33.111136 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108206    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:33.111136 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108210    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:33.111136 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108212    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:33.111136 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108215    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:33.111136 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.108218    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:33.111136 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109508    2570 flags.go:64] FLAG: --address="0.0.0.0"
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109518    2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109525    2570 flags.go:64] FLAG: --anonymous-auth="true"
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109530    2570 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109535    2570 flags.go:64] FLAG: --authentication-token-webhook="false"
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109538    2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109543    2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109548    2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109551    2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109554    2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109559    2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109562    2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109565    2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109569    2570 flags.go:64] FLAG: --cgroup-root=""
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109572    2570 flags.go:64] FLAG: --cgroups-per-qos="true"
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109575    2570 flags.go:64] FLAG: --client-ca-file=""
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109578    2570 flags.go:64] FLAG: --cloud-config=""
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109581    2570 flags.go:64] FLAG: --cloud-provider="external"
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109583    2570 flags.go:64] FLAG: --cluster-dns="[]"
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109589    2570 flags.go:64] FLAG: --cluster-domain=""
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109591    2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109594    2570 flags.go:64] FLAG: --config-dir=""
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109597    2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109601    2570 flags.go:64] FLAG: --container-log-max-files="5"
May 20 13:50:33.111664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109604    2570 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109622    2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109626    2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109629    2570 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109632    2570 flags.go:64] FLAG: --contention-profiling="false"
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109635    2570 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109638    2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109641    2570 flags.go:64] FLAG: --cpu-manager-policy="none"
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109644    2570 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109648    2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109651    2570 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109655    2570 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109658    2570 flags.go:64] FLAG: --enable-load-reader="false"
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109661    2570 flags.go:64] FLAG: --enable-server="true"
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109665    2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109670    2570 flags.go:64] FLAG: --event-burst="100"
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109673    2570 flags.go:64] FLAG: --event-qps="50"
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109676    2570 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109680    2570 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109683    2570 flags.go:64] FLAG: --eviction-hard=""
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109687    2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109690    2570 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109693    2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109696    2570 flags.go:64] FLAG: --eviction-soft=""
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109698    2570 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 20 13:50:33.112231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109701    2570 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109704    2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109707    2570 flags.go:64] FLAG: --experimental-mounter-path=""
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109710    2570 flags.go:64] FLAG: --fail-cgroupv1="false"
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109712    2570 flags.go:64] FLAG: --fail-swap-on="true"
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109715    2570 flags.go:64] FLAG: --feature-gates=""
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109719    2570 flags.go:64] FLAG: --file-check-frequency="20s"
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109722    2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109725    2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109729    2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109732    2570 flags.go:64] FLAG: --healthz-port="10248"
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109735    2570 flags.go:64] FLAG: --help="false"
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109738    2570 flags.go:64] FLAG: --hostname-override="ip-10-0-136-75.ec2.internal"
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109741    2570 flags.go:64] FLAG: --housekeeping-interval="10s"
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109744    2570 flags.go:64] FLAG: --http-check-frequency="20s"
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109746    2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109750    2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109753    2570 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109757    2570 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109760    2570 flags.go:64] FLAG: --image-service-endpoint=""
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109763    2570 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109765    2570 flags.go:64] FLAG: --kube-api-burst="100"
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109768    2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109771    2570 flags.go:64] FLAG: --kube-api-qps="50"
May 20 13:50:33.112859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109774    2570 flags.go:64] FLAG: --kube-reserved=""
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109777    2570 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109780    2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109783    2570 flags.go:64] FLAG: --kubelet-cgroups=""
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109785    2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109788    2570 flags.go:64] FLAG: --lock-file=""
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109791    2570 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109793    2570 flags.go:64] FLAG: --log-flush-frequency="5s"
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109796    2570 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109801    2570 flags.go:64] FLAG: --log-json-split-stream="false"
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109804    2570 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109807    2570 flags.go:64] FLAG: --log-text-split-stream="false"
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109810    2570 flags.go:64] FLAG: --logging-format="text"
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109813    2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109817    2570 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109819    2570 flags.go:64] FLAG: --manifest-url=""
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109822    2570 flags.go:64] FLAG: --manifest-url-header=""
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109826    2570 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109830    2570 flags.go:64] FLAG: --max-open-files="1000000"
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109834    2570 flags.go:64] FLAG: --max-pods="110"
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109837    2570 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109840    2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109842    2570 flags.go:64] FLAG: --memory-manager-policy="None"
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109845    2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109848    2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 20 13:50:33.113417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109851    2570 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109854    2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109862    2570 flags.go:64] FLAG: --node-status-max-images="50"
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109865    2570 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109868    2570 flags.go:64] FLAG: --oom-score-adj="-999"
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109871    2570 flags.go:64] FLAG: --pod-cidr=""
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109874    2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109879    2570 flags.go:64] FLAG: --pod-manifest-path=""
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109882    2570 flags.go:64] FLAG: --pod-max-pids="-1"
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109885    2570 flags.go:64] FLAG: --pods-per-core="0"
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109888    2570 flags.go:64] FLAG: --port="10250"
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109891    2570 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109894    2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06074411d43b3f49f"
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109897    2570 flags.go:64] FLAG: --qos-reserved=""
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109900    2570 flags.go:64] FLAG: --read-only-port="10255"
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109903    2570 flags.go:64] FLAG: --register-node="true"
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109906    2570 flags.go:64] FLAG: --register-schedulable="true"
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109909    2570 flags.go:64] FLAG: --register-with-taints=""
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109913    2570 flags.go:64] FLAG: --registry-burst="10"
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109915    2570 flags.go:64] FLAG: --registry-qps="5"
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109918    2570 flags.go:64] FLAG: --reserved-cpus=""
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109921    2570 flags.go:64] FLAG: --reserved-memory=""
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109925    2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109928    2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109931    2570 flags.go:64] FLAG: --rotate-certificates="false"
May 20 13:50:33.114021 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109934    2570 flags.go:64] FLAG: --rotate-server-certificates="false"
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109937    2570 flags.go:64] FLAG: --runonce="false"
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109940    2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109944    2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109946    2570 flags.go:64] FLAG: --seccomp-default="false"
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109949    2570 flags.go:64] FLAG: --serialize-image-pulls="true"
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109951    2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109954    2570 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109957    2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109960    2570 flags.go:64] FLAG: --storage-driver-password="root"
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109963    2570 flags.go:64] FLAG: --storage-driver-secure="false"
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109966    2570 flags.go:64] FLAG: --storage-driver-table="stats"
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109969    2570 flags.go:64] FLAG: --storage-driver-user="root"
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109972    2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109974    2570 flags.go:64] FLAG: --sync-frequency="1m0s"
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109977    2570 flags.go:64] FLAG: --system-cgroups=""
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109980    2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109985    2570 flags.go:64] FLAG: --system-reserved-cgroup=""
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109988    2570 flags.go:64] FLAG: --tls-cert-file=""
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109991    2570 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109996    2570 flags.go:64] FLAG: --tls-min-version=""
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.109998    2570 flags.go:64] FLAG: --tls-private-key-file=""
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.110001    2570 flags.go:64] FLAG: --topology-manager-policy="none"
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.110003    2570 flags.go:64] FLAG: --topology-manager-policy-options=""
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.110006    2570 flags.go:64] FLAG: --topology-manager-scope="container"
May 20 13:50:33.114629 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.110009    2570 flags.go:64] FLAG: --v="2"
May 20 13:50:33.115200 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.110013    2570 flags.go:64] FLAG: --version="false"
May 20 13:50:33.115200 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.110017    2570 flags.go:64] FLAG: --vmodule=""
May 20 13:50:33.115200 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.110021    2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 20 13:50:33.115200 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.110024    2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 20 13:50:33.115200 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110138    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:33.115200 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110141    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:33.115200 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110145    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:33.115200 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110149    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:33.115200 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110152    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:33.115200 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110156    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:33.115200 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110158    2570 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:33.115200 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110161    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:33.115200 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110164    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:33.115200 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110167    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:33.115200 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110169    2570 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:33.115200 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110172    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:33.115200 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110175    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:33.115200 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110177    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:33.115200 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110180    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:33.115200 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110183    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:33.115928 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110185    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:33.115928 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110188    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:33.115928 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110190    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:33.115928 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110193    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:33.115928 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110195    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:33.115928 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110198    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:33.115928 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110200    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:33.115928 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110203    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:33.115928 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110205    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:33.115928 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110208    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:33.115928 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110211    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:33.115928 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110213    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:33.115928 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110216    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:33.115928 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110220    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:33.115928 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110223    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:33.115928 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110226    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:33.115928 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110229    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:33.115928 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110232    2570 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:33.115928 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110237    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:33.116831 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110240    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:33.116831 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110243    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:33.116831 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110246    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:33.116831 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110249    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:33.116831 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110252    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:33.116831 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110255    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:33.116831 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110258    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:33.116831 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110262    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:33.116831 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110264    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:33.116831 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110267    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:33.116831 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110269    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:33.116831 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110272    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:33.116831 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110274    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:33.116831 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110277    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:33.116831 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110279    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:33.116831 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110282    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:33.116831 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110284    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:33.116831 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110287    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:33.116831 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110289    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:33.116831 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110292    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:33.117674 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110295    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:33.117674 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110297    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:33.117674 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110300    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:33.117674 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110302    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:33.117674 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110305    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:33.117674 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110308    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:33.117674 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110310    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:33.117674 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110313    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:33.117674 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110315    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:33.117674 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110318    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:33.117674 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110320    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:33.117674 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110324    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:33.117674 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110326    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:33.117674 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110329    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:33.117674 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110331    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:33.117674 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110335    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:33.117674 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110339    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:33.117674 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110341    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:33.117674 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110344    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:33.117674 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110348    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:33.118520 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110350    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:33.118520 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110353    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:33.118520 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110355    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:33.118520 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110358    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:33.118520 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110360    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:33.118520 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110362    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:33.118520 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110365    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:33.118520 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110367    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:33.118520 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110370    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:33.118520 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110372    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:33.118520 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.110375    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:33.118520 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.110379    2570 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 13:50:33.118520 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.117504    2570 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 20 13:50:33.118520 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.117528    2570 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 20 13:50:33.118520 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117597    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:33.118520 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117605    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:33.119012 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117630    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:33.119012 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117635    2570 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:33.119012 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117639    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:33.119012 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117643    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:33.119012 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117648    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:33.119012 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117652    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:33.119012 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117656    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:33.119012 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117660    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:33.119012 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117664    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:33.119012 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117669    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:33.119012 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117674    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:33.119012 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117678    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:33.119012 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117682    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:33.119012 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117686    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:33.119012 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117690    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:33.119012 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117694    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:33.119012 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117698    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:33.119012 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117702    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:33.119012 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117707    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:33.119012 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117711    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:33.119606 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117714    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:33.119606 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117719    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:33.119606 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117724    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:33.119606 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117729    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:33.119606 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117733    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:33.119606 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117737    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:33.119606 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117741    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:33.119606 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117746    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:33.119606 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117750    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:33.119606 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117760    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:33.119606 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117765    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:33.119606 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117769    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:33.119606 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117774    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:33.119606 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117778    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:33.119606 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117783    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:33.119606 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117786    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:33.119606 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117790    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:33.119606 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117794    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:33.119606 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117799    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:33.120263 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117805    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:33.120263 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117811    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:33.120263 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117816    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:33.120263 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117820    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:33.120263 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117824    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:33.120263 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117830    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:33.120263 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117837    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:33.120263 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117842    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:33.120263 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117846    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:33.120263 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117851    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:33.120263 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117856    2570 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:33.120263 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117860    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:33.120263 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117865    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:33.120263 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117870    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:33.120263 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117875    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:33.120263 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117879    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:33.120263 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117883    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:33.120263 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117887    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:33.120263 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117891    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:33.120992 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117895    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:33.120992 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117899    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:33.120992 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117904    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:33.120992 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117909    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:33.120992 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117915    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:33.120992 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117920    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:33.120992 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117924    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:33.120992 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117929    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:33.120992 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117933    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:33.120992 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117937    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:33.120992 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117940    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:33.120992 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117945    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:33.120992 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117949    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:33.120992 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117952    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:33.120992 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117956    2570 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:33.120992 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117960    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:33.120992 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117964    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:33.120992 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117968    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:33.120992 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117972    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:33.120992 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117976    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:33.121492 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117980    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:33.121492 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117984    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:33.121492 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117988    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:33.121492 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117992    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:33.121492 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.117997    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:33.121492 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118001    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:33.121492 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.118008    2570 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 13:50:33.121492 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118165    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:33.121492 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118173    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:33.121492 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118177    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:33.121492 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118181    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:33.121492 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118186    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:33.121492 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118191    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:33.121492 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118194    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:33.121492 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118199    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:33.121864 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118203    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:33.121864 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118207    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:33.121864 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118212    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:33.121864 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118216    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:33.121864 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118220    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:33.121864 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118225    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:33.121864 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118228    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:33.121864 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118233    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:33.121864 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118241    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:33.121864 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118245    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:33.121864 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118249    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:33.121864 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118253    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:33.121864 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118257    2570 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:33.121864 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118262    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:33.121864 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118266    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:33.121864 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118269    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:33.121864 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118274    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:33.121864 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118278    2570 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:33.121864 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118282    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:33.121864 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118286    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:33.122460 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118291    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:33.122460 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118295    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:33.122460 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118299    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:33.122460 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118303    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:33.122460 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118308    2570 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:33.122460 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118312    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:33.122460 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118315    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:33.122460 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118319    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:33.122460 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118324    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:33.122460 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118328    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:33.122460 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118333    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:33.122460 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118337    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:33.122460 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118340    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:33.122460 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118345    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:33.122460 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118349    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:33.122460 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118353    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:33.122460 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118358    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:33.122460 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118362    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:33.122460 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118366    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:33.122460 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118370    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:33.123225 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118374    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:33.123225 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118378    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:33.123225 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118382    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:33.123225 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118387    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:33.123225 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118391    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:33.123225 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118394    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:33.123225 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118398    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:33.123225 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118402    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:33.123225 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118406    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:33.123225 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118411    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:33.123225 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118414    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:33.123225 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118418    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:33.123225 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118422    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:33.123225 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118426    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:33.123225 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118430    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:33.123225 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118434    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:33.123225 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118439    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:33.123225 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118443    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:33.123225 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118447    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:33.123225 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118450    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:33.124034 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118454    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:33.124034 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118460    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:33.124034 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118466    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:33.124034 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118471    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:33.124034 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118475    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:33.124034 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118479    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:33.124034 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118483    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:33.124034 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118487    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:33.124034 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118492    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:33.124034 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118496    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:33.124034 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118500    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:33.124034 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118504    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:33.124034 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118508    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:33.124034 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118512    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:33.124034 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118516    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:33.124034 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118520    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:33.124034 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118524    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:33.124034 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:33.118527    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:33.124630 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.118534    2570 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 13:50:33.124630 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.119371    2570 server.go:962] "Client rotation is on, will bootstrap in background"
May 20 13:50:33.124630 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.124593    2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 20 13:50:33.125770 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.125757    2570 server.go:1019] "Starting client certificate rotation"
May 20 13:50:33.125875 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.125858    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 20 13:50:33.125928 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.125911    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 20 13:50:33.151993 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.151959    2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 20 13:50:33.155682 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.155656    2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 20 13:50:33.173501 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.173477    2570 log.go:25] "Validated CRI v1 runtime API"
May 20 13:50:33.178820 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.178801    2570 log.go:25] "Validated CRI v1 image API"
May 20 13:50:33.180692 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.180672    2570 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 20 13:50:33.187511 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.187483    2570 fs.go:135] Filesystem UUIDs: map[05c8f809-de21-4e7f-8b66-33cf9197d333:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 aa255bb5-c00d-4237-b53e-613c130c7afe:/dev/nvme0n1p4]
May 20 13:50:33.187658 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.187510    2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 20 13:50:33.188399 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.188379    2570 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 20 13:50:33.192901 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.192743    2570 manager.go:217] Machine: {Timestamp:2026-05-20 13:50:33.192182946 +0000 UTC m=+0.429799584 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2499998 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c60b192878d65637c3d3f76e0341e SystemUUID:ec2c60b1-9287-8d65-637c-3d3f76e0341e BootID:6f077f17-d3ac-4fa2-9acc-071ef3cbd57a Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:bb:f1:4e:e3:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:bb:f1:4e:e3:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:79:90:92:1a:ef Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 20 13:50:33.193626 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.193596    2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 20 13:50:33.193746 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.193730    2570 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 20 13:50:33.195810 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.195783    2570 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 20 13:50:33.195957 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.195814    2570 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-75.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 20 13:50:33.196009 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.195967    2570 topology_manager.go:138] "Creating topology manager with none policy"
May 20 13:50:33.196009 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.195976    2570 container_manager_linux.go:306] "Creating device plugin manager"
May 20 13:50:33.196009 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.195984    2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 20 13:50:33.196009 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.195997    2570 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 20 13:50:33.197211 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.197200    2570 state_mem.go:36] "Initialized new in-memory state store"
May 20 13:50:33.197496 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.197487    2570 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 20 13:50:33.199899 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.199888    2570 kubelet.go:491] "Attempting to sync node with API server"
May 20 13:50:33.199933 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.199904    2570 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 20 13:50:33.199933 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.199916    2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 20 13:50:33.199933 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.199924    2570 kubelet.go:397] "Adding apiserver pod source"
May 20 13:50:33.199933 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.199934    2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 20 13:50:33.201072 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.201060    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 20 13:50:33.201107 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.201079    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 20 13:50:33.203889 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.203873    2570 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 20 13:50:33.205684 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.205670    2570 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 20 13:50:33.206880 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.206865    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 20 13:50:33.206925 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.206888    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 20 13:50:33.206925 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.206897    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 20 13:50:33.206925 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.206905    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 20 13:50:33.206925 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.206918    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 20 13:50:33.207031 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.206927    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 20 13:50:33.207031 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.206936    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 20 13:50:33.207031 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.206943    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 20 13:50:33.207031 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.206953    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 20 13:50:33.207031 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.206961    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 20 13:50:33.207031 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.206973    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 20 13:50:33.207031 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.206983    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 20 13:50:33.208598 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.208587    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 20 13:50:33.208654 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.208601    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 20 13:50:33.212292 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.212277    2570 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 20 13:50:33.212372 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.212316    2570 server.go:1295] "Started kubelet"
May 20 13:50:33.212449 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.212411    2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 20 13:50:33.212546 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.212473    2570 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 20 13:50:33.212581 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.212543    2570 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-75.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 20 13:50:33.212581 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.212571    2570 server_v1.go:47] "podresources" method="list" useActivePods=true
May 20 13:50:33.213644 ip-10-0-136-75 systemd[1]: Started Kubernetes Kubelet.
May 20 13:50:33.214154 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.213731    2570 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 20 13:50:33.214550 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:33.214436    2570 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 20 13:50:33.214550 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:33.214436    2570 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-75.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 20 13:50:33.214951 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.214928    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2c7bs"
May 20 13:50:33.215248 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.215235    2570 server.go:317] "Adding debug handlers to kubelet server"
May 20 13:50:33.219301 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.219281    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 20 13:50:33.219760 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.219738    2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 20 13:50:33.219884 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.219870    2570 volume_manager.go:295] "The desired_state_of_world populator starts"
May 20 13:50:33.219951 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.219885    2570 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 20 13:50:33.219951 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.219937    2570 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 20 13:50:33.220060 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.220042    2570 reconstruct.go:97] "Volume reconstruction finished"
May 20 13:50:33.220113 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.220072    2570 reconciler.go:26] "Reconciler: start to sync state"
May 20 13:50:33.220496 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.220480    2570 factory.go:55] Registering systemd factory
May 20 13:50:33.220496 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.220498    2570 factory.go:223] Registration of the systemd container factory successfully
May 20 13:50:33.220795 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:33.220777    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found"
May 20 13:50:33.220997 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.220985    2570 factory.go:153] Registering CRI-O factory
May 20 13:50:33.221086 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.221073    2570 factory.go:223] Registration of the crio container factory successfully
May 20 13:50:33.223061 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:33.220769    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-75.ec2.internal.18b14a4f86c9a64d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-75.ec2.internal,UID:ip-10-0-136-75.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-75.ec2.internal,},FirstTimestamp:2026-05-20 13:50:33.212290637 +0000 UTC m=+0.449907274,LastTimestamp:2026-05-20 13:50:33.212290637 +0000 UTC m=+0.449907274,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-75.ec2.internal,}"
May 20 13:50:33.223161 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.223095    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2c7bs"
May 20 13:50:33.223161 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.223127    2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 20 13:50:33.223241 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.223193    2570 factory.go:103] Registering Raw factory
May 20 13:50:33.223241 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.223210    2570 manager.go:1196] Started watching for new ooms in manager
May 20 13:50:33.223778 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.223759    2570 manager.go:319] Starting recovery of all containers
May 20 13:50:33.228489 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:33.228465    2570 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 20 13:50:33.231906 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.231721    2570 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:33.233866 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:33.233846    2570 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-75.ec2.internal\" not found" node="ip-10-0-136-75.ec2.internal"
May 20 13:50:33.235341 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.235324    2570 manager.go:324] Recovery completed
May 20 13:50:33.236860 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:33.236836    2570 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
May 20 13:50:33.239900 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.239883    2570 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:33.241864 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.241849    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:33.241922 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.241878    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:33.241922 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.241889    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:33.242386 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.242373    2570 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 20 13:50:33.242386 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.242385    2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 20 13:50:33.242480 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.242406    2570 state_mem.go:36] "Initialized new in-memory state store"
May 20 13:50:33.244600 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.244588    2570 policy_none.go:49] "None policy: Start"
May 20 13:50:33.244658 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.244605    2570 memory_manager.go:186] "Starting memorymanager" policy="None"
May 20 13:50:33.244658 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.244635    2570 state_mem.go:35] "Initializing new in-memory state store"
May 20 13:50:33.291286 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.291271    2570 manager.go:341] "Starting Device Plugin manager"
May 20 13:50:33.315834 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:33.291303    2570 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 20 13:50:33.315834 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.291314    2570 server.go:85] "Starting device plugin registration server"
May 20 13:50:33.315834 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.291545    2570 eviction_manager.go:189] "Eviction manager: starting control loop"
May 20 13:50:33.315834 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.291575    2570 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 20 13:50:33.315834 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.291686    2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 20 13:50:33.315834 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.291754    2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 20 13:50:33.315834 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.291762    2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 20 13:50:33.315834 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:33.292875    2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 20 13:50:33.315834 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:33.292916    2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-75.ec2.internal\" not found"
May 20 13:50:33.357249 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.357156    2570 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 20 13:50:33.358690 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.358673    2570 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 20 13:50:33.358776 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.358699    2570 status_manager.go:230] "Starting to sync pod status with apiserver"
May 20 13:50:33.358776 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.358719    2570 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 20 13:50:33.358776 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.358726    2570 kubelet.go:2451] "Starting kubelet main sync loop"
May 20 13:50:33.358776 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:33.358763    2570 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 20 13:50:33.363191 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.363174    2570 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:33.392406 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.392368    2570 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:33.393523 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.393498    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:33.393603 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.393531    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:33.393603 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.393543    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:33.393603 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.393568    2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-75.ec2.internal"
May 20 13:50:33.402365 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.402346    2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-75.ec2.internal"
May 20 13:50:33.402411 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:33.402372    2570 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-75.ec2.internal\": node \"ip-10-0-136-75.ec2.internal\" not found"
May 20 13:50:33.420468 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:33.420440    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found"
May 20 13:50:33.459881 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.459843    2570 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-75.ec2.internal"]
May 20 13:50:33.460013 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.459923    2570 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:33.461441 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.461424    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:33.461528 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.461457    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:33.461528 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.461470    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:33.463533 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.463510    2570 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:33.463754 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.463739    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal"
May 20 13:50:33.463807 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.463768    2570 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:33.464321 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.464305    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:33.464427 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.464329    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:33.464427 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.464348    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:33.464427 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.464358    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:33.464427 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.464334    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:33.464427 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.464412    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:33.466237 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.466219    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-75.ec2.internal"
May 20 13:50:33.466306 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.466247    2570 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:33.466924 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.466910    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:33.467002 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.466935    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:33.467002 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.466949    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:33.490252 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:33.490232    2570 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-75.ec2.internal\" not found" node="ip-10-0-136-75.ec2.internal"
May 20 13:50:33.494022 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:33.494005    2570 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-75.ec2.internal\" not found" node="ip-10-0-136-75.ec2.internal"
May 20 13:50:33.520958 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:33.520937    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found"
May 20 13:50:33.522090 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.522063    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/38a3d6bde081a3bde4f3c1a1f8fce848-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal\" (UID: \"38a3d6bde081a3bde4f3c1a1f8fce848\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal"
May 20 13:50:33.522090 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.522090    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38a3d6bde081a3bde4f3c1a1f8fce848-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal\" (UID: \"38a3d6bde081a3bde4f3c1a1f8fce848\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal"
May 20 13:50:33.522235 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.522109    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6959407f02e6e503e019a9457b3f87d1-config\") pod \"kube-apiserver-proxy-ip-10-0-136-75.ec2.internal\" (UID: \"6959407f02e6e503e019a9457b3f87d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-75.ec2.internal"
May 20 13:50:33.621919 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:33.621832    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found"
May 20 13:50:33.622980 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.622961    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/38a3d6bde081a3bde4f3c1a1f8fce848-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal\" (UID: \"38a3d6bde081a3bde4f3c1a1f8fce848\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal"
May 20 13:50:33.623040 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.622989    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38a3d6bde081a3bde4f3c1a1f8fce848-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal\" (UID: \"38a3d6bde081a3bde4f3c1a1f8fce848\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal"
May 20 13:50:33.623040 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.623006    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6959407f02e6e503e019a9457b3f87d1-config\") pod \"kube-apiserver-proxy-ip-10-0-136-75.ec2.internal\" (UID: \"6959407f02e6e503e019a9457b3f87d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-75.ec2.internal"
May 20 13:50:33.623130 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.623054    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6959407f02e6e503e019a9457b3f87d1-config\") pod \"kube-apiserver-proxy-ip-10-0-136-75.ec2.internal\" (UID: \"6959407f02e6e503e019a9457b3f87d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-75.ec2.internal"
May 20 13:50:33.623130 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.623079    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38a3d6bde081a3bde4f3c1a1f8fce848-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal\" (UID: \"38a3d6bde081a3bde4f3c1a1f8fce848\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal"
May 20 13:50:33.623130 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.623079    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/38a3d6bde081a3bde4f3c1a1f8fce848-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal\" (UID: \"38a3d6bde081a3bde4f3c1a1f8fce848\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal"
May 20 13:50:33.722593 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:33.722548    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found"
May 20 13:50:33.792869 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.792834    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal"
May 20 13:50:33.797651 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:33.797635    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-75.ec2.internal"
May 20 13:50:33.822896 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:33.822861    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found"
May 20 13:50:33.923359 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:33.923258    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found"
May 20 13:50:34.023680 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:34.023648    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found"
May 20 13:50:34.124100 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:34.124066    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found"
May 20 13:50:34.125150 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:34.125140    2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 20 13:50:34.125307 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:34.125291    2570 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 20 13:50:34.125346 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:34.125319    2570 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 20 13:50:34.219760 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:34.219679    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 20 13:50:34.224951 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:34.224920    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-19 13:45:33 +0000 UTC" deadline="2027-10-25 08:35:56.527492248 +0000 UTC"
May 20 13:50:34.224951 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:34.224952    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12546h45m22.302543656s"
May 20 13:50:34.225096 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:34.224929    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found"
May 20 13:50:34.234781 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:34.234759    2570 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 20 13:50:34.279069 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:34.279042    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nv6nb"
May 20 13:50:34.291128 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:34.291097    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nv6nb"
May 20 13:50:34.325829 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:34.325800    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found"
May 20 13:50:34.426262 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:34.426232    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found"
May 20 13:50:34.427994 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:34.427950    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a3d6bde081a3bde4f3c1a1f8fce848.slice/crio-46c2f76337e41e6cf5a07491027d88bbfc88cc806f4a40e8e0c2154a7e8be0fe WatchSource:0}: Error finding container 46c2f76337e41e6cf5a07491027d88bbfc88cc806f4a40e8e0c2154a7e8be0fe: Status 404 returned error can't find the container with id 46c2f76337e41e6cf5a07491027d88bbfc88cc806f4a40e8e0c2154a7e8be0fe
May 20 13:50:34.428266 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:34.428246    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6959407f02e6e503e019a9457b3f87d1.slice/crio-8898992cf14ffb508afed109d5ebd197dc95ac0e1f2e5a2bb636d0e60bbc6599 WatchSource:0}: Error finding container 8898992cf14ffb508afed109d5ebd197dc95ac0e1f2e5a2bb636d0e60bbc6599: Status 404 returned error can't find the container with id 8898992cf14ffb508afed109d5ebd197dc95ac0e1f2e5a2bb636d0e60bbc6599
May 20 13:50:34.431602 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:34.431589    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 13:50:34.461700 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:34.460460    2570 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:34.521025 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:34.520995    2570 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal"
May 20 13:50:34.536755 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:34.536729    2570 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:34.557898 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:34.557875    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 20 13:50:34.559022 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:34.559009    2570 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-75.ec2.internal"
May 20 13:50:34.575891 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:34.575871    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 20 13:50:35.201588 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.201332    2570 apiserver.go:52] "Watching apiserver"
May 20 13:50:35.208029 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.208004    2570 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 20 13:50:35.208421 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.208394    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz","openshift-dns/node-resolver-cs8gn","openshift-image-registry/node-ca-dk92v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal","openshift-multus/multus-9wtjr","openshift-network-diagnostics/network-check-target-clw2n","openshift-ovn-kubernetes/ovnkube-node-kfhg5","kube-system/konnectivity-agent-x6mng","kube-system/kube-apiserver-proxy-ip-10-0-136-75.ec2.internal","openshift-cluster-node-tuning-operator/tuned-jt2w6","openshift-multus/multus-additional-cni-plugins-zgj4t","openshift-multus/network-metrics-daemon-rhcx7","openshift-network-operator/iptables-alerter-bwkmk"]
May 20 13:50:35.210865 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.210844    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.213742 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.213721    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cs8gn"
May 20 13:50:35.213860 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.213837    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 20 13:50:35.213979 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.213727    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 20 13:50:35.214038 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.213996    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dlgjv\""
May 20 13:50:35.215920 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.215903    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 20 13:50:35.216710 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.216395    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 20 13:50:35.216710 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.216430    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ntfbn\""
May 20 13:50:35.216710 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.216432    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dk92v"
May 20 13:50:35.219638 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.219035    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.221444 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.221394    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 20 13:50:35.222122 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.222097    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-96zns\""
May 20 13:50:35.222473 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.222456    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 20 13:50:35.222749 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.222732    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 20 13:50:35.222977 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.222959    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hnfhd\""
May 20 13:50:35.223063 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.223033    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 20 13:50:35.223371 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.223355    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 20 13:50:35.223601 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.223586    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 20 13:50:35.224307 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.224287    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.225822 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.225802    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 20 13:50:35.227483 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.227460    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 20 13:50:35.228046 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.228019    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:50:35.228143 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:35.228104    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clw2n" podUID="31842022-6a01-49b0-8b26-fb0d882cdf6f"
May 20 13:50:35.228143 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.228128    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x6mng"
May 20 13:50:35.229180 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.228893    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 20 13:50:35.229354 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.229334    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pdwk9\""
May 20 13:50:35.229522 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.229509    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 20 13:50:35.229724 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.229707    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 20 13:50:35.229835 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.229774    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 20 13:50:35.229901 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.229881    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 20 13:50:35.230483 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.230465    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7jcz2\""
May 20 13:50:35.230579 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.230558    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 20 13:50:35.230837 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.230697    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 20 13:50:35.231011 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.230977    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.231098 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231021    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-lib-modules\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.231098 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231042    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4bdf5520-b68e-4063-b150-50ff558b61e2-etc-tuned\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.231098 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231058    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b985f05d-7aa5-42b7-a5d7-3b18a2769356-hosts-file\") pod \"node-resolver-cs8gn\" (UID: \"b985f05d-7aa5-42b7-a5d7-3b18a2769356\") " pod="openshift-dns/node-resolver-cs8gn"
May 20 13:50:35.231098 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231071    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-multus-cni-dir\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.231098 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231093    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpd62\" (UniqueName: \"kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62\") pod \"network-check-target-clw2n\" (UID: \"31842022-6a01-49b0-8b26-fb0d882cdf6f\") " pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:50:35.231281 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231113    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-etc-modprobe-d\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.231281 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231156    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-sys\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.231281 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231225    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bdf5520-b68e-4063-b150-50ff558b61e2-tmp\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.231281 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231256    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh895\" (UniqueName: \"kubernetes.io/projected/efd2c3ea-0867-4025-b5c6-03b9a0d744c5-kube-api-access-xh895\") pod \"node-ca-dk92v\" (UID: \"efd2c3ea-0867-4025-b5c6-03b9a0d744c5\") " pod="openshift-image-registry/node-ca-dk92v"
May 20 13:50:35.231431 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231281    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-os-release\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.231431 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231307    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-host-var-lib-cni-bin\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.231431 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231330    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-hostroot\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.231431 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231354    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-multus-daemon-config\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.231431 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231380    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-var-lib-kubelet\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.231431 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231405    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-cnibin\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.231431 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231422    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-multus-socket-dir-parent\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.231684 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231438    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-host-var-lib-cni-multus\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.231684 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231465    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-host-run-multus-certs\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.231684 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231487    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ktv2\" (UniqueName: \"kubernetes.io/projected/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-kube-api-access-5ktv2\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.231684 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231506    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-etc-sysconfig\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.231684 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231535    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-multus-conf-dir\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.231684 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231593    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-etc-kubernetes\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.231684 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231642    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-etc-sysctl-conf\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.231684 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231668    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-host\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.231955 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231691    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-run\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.231955 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231714    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjjj4\" (UniqueName: \"kubernetes.io/projected/b985f05d-7aa5-42b7-a5d7-3b18a2769356-kube-api-access-tjjj4\") pod \"node-resolver-cs8gn\" (UID: \"b985f05d-7aa5-42b7-a5d7-3b18a2769356\") " pod="openshift-dns/node-resolver-cs8gn"
May 20 13:50:35.231955 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231739    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/efd2c3ea-0867-4025-b5c6-03b9a0d744c5-serviceca\") pod \"node-ca-dk92v\" (UID: \"efd2c3ea-0867-4025-b5c6-03b9a0d744c5\") " pod="openshift-image-registry/node-ca-dk92v"
May 20 13:50:35.231955 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231760    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-host-run-netns\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.231955 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231782    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-host-var-lib-kubelet\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.231955 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231828    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-etc-sysctl-d\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.231955 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231861    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-etc-systemd\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.231955 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231886    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mxzt\" (UniqueName: \"kubernetes.io/projected/4bdf5520-b68e-4063-b150-50ff558b61e2-kube-api-access-2mxzt\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.231955 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231910    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-system-cni-dir\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.231955 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231934    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-cni-binary-copy\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.232273 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231958    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-etc-kubernetes\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.232273 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.231988    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b985f05d-7aa5-42b7-a5d7-3b18a2769356-tmp-dir\") pod \"node-resolver-cs8gn\" (UID: \"b985f05d-7aa5-42b7-a5d7-3b18a2769356\") " pod="openshift-dns/node-resolver-cs8gn"
May 20 13:50:35.232273 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.232014    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/efd2c3ea-0867-4025-b5c6-03b9a0d744c5-host\") pod \"node-ca-dk92v\" (UID: \"efd2c3ea-0867-4025-b5c6-03b9a0d744c5\") " pod="openshift-image-registry/node-ca-dk92v"
May 20 13:50:35.232273 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.232037    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-host-run-k8s-cni-cncf-io\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.233144 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.233127    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.235546 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.235512    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:35.235673 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:35.235652    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhcx7" podUID="6969002d-0b97-4cb6-ae5b-03d5ef9dddf8"
May 20 13:50:35.236632 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.236596    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 20 13:50:35.236724 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.236692    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qspn5\""
May 20 13:50:35.236724 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.236709    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 20 13:50:35.236834 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.236643    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 20 13:50:35.236834 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.236605    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 20 13:50:35.236952 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.236937    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 20 13:50:35.237602 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.237444    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4gvkl\""
May 20 13:50:35.238539 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.238186    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bwkmk"
May 20 13:50:35.240470 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.240453    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 20 13:50:35.240771 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.240574    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 20 13:50:35.240979 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.240826    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mkxvw\""
May 20 13:50:35.240979 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.240915    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 20 13:50:35.292334 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.292300    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 13:45:34 +0000 UTC" deadline="2028-01-22 02:03:21.767048718 +0000 UTC"
May 20 13:50:35.292334 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.292326    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14676h12m46.474725623s"
May 20 13:50:35.321389 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.321349    2570 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 20 13:50:35.333061 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333029    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-run\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.333245 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333077    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-host-run-netns\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.333245 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333103    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-host-var-lib-kubelet\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.333245 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333138    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrl92\" (UniqueName: \"kubernetes.io/projected/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-kube-api-access-mrl92\") pod \"network-metrics-daemon-rhcx7\" (UID: \"6969002d-0b97-4cb6-ae5b-03d5ef9dddf8\") " pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:35.333245 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333156    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-node-log\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.333245 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333166    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-run\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.333245 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333170    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-host-cni-netd\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.333245 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333167    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-host-run-netns\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.333480 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333244    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-host-var-lib-kubelet\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.333480 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333280    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-954t2\" (UniqueName: \"kubernetes.io/projected/3142b56b-8ab7-4799-8b7a-46420a701680-kube-api-access-954t2\") pod \"iptables-alerter-bwkmk\" (UID: \"3142b56b-8ab7-4799-8b7a-46420a701680\") " pod="openshift-network-operator/iptables-alerter-bwkmk"
May 20 13:50:35.333480 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333326    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-etc-sysctl-d\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.333480 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333360    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-system-cni-dir\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.333480 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333390    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-host-cni-bin\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.333480 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333417    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l2gz\" (UniqueName: \"kubernetes.io/projected/56992f6a-130b-435f-a75b-bf6b103ec8e1-kube-api-access-8l2gz\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.333480 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333432    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-system-cni-dir\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.333480 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333438    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-etc-sysctl-d\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.333480 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333446    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/efd2c3ea-0867-4025-b5c6-03b9a0d744c5-host\") pod \"node-ca-dk92v\" (UID: \"efd2c3ea-0867-4025-b5c6-03b9a0d744c5\") " pod="openshift-image-registry/node-ca-dk92v"
May 20 13:50:35.333480 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333475    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/37b49aa0-a7ad-4499-9989-3bfc30ddf3ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vfvkz\" (UID: \"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.333871 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333499    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/efd2c3ea-0867-4025-b5c6-03b9a0d744c5-host\") pod \"node-ca-dk92v\" (UID: \"efd2c3ea-0867-4025-b5c6-03b9a0d744c5\") " pod="openshift-image-registry/node-ca-dk92v"
May 20 13:50:35.333871 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333533    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/56992f6a-130b-435f-a75b-bf6b103ec8e1-ovnkube-config\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.333871 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333555    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-lib-modules\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.333871 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333572    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4bdf5520-b68e-4063-b150-50ff558b61e2-etc-tuned\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.333871 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333596    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b985f05d-7aa5-42b7-a5d7-3b18a2769356-hosts-file\") pod \"node-resolver-cs8gn\" (UID: \"b985f05d-7aa5-42b7-a5d7-3b18a2769356\") " pod="openshift-dns/node-resolver-cs8gn"
May 20 13:50:35.333871 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333635    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-multus-cni-dir\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.333871 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333657    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/37b49aa0-a7ad-4499-9989-3bfc30ddf3ee-device-dir\") pod \"aws-ebs-csi-driver-node-vfvkz\" (UID: \"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.333871 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333676    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-sys\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.333871 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333689    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b985f05d-7aa5-42b7-a5d7-3b18a2769356-hosts-file\") pod \"node-resolver-cs8gn\" (UID: \"b985f05d-7aa5-42b7-a5d7-3b18a2769356\") " pod="openshift-dns/node-resolver-cs8gn"
May 20 13:50:35.333871 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333699    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-host-var-lib-cni-bin\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.333871 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333728    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-hostroot\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.333871 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333732    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-lib-modules\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.333871 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333743    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-host-var-lib-cni-bin\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.333871 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333747    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-host-run-multus-certs\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.333871 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333768    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/37b49aa0-a7ad-4499-9989-3bfc30ddf3ee-sys-fs\") pod \"aws-ebs-csi-driver-node-vfvkz\" (UID: \"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.333871 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333791    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-sys\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.333871 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333796    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-multus-cni-dir\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.333871 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333788    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ead02d78-f55b-4612-bb8d-057d70f6ecc4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.334633 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333828    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-host-run-multus-certs\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.334633 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333829    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-systemd-units\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.334633 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333853    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-hostroot\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.334633 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333854    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-run-systemd\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.334633 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333879    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-multus-socket-dir-parent\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.334633 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333906    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-host-var-lib-cni-multus\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.334633 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333924    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs\") pod \"network-metrics-daemon-rhcx7\" (UID: \"6969002d-0b97-4cb6-ae5b-03d5ef9dddf8\") " pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:35.334633 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333941    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3142b56b-8ab7-4799-8b7a-46420a701680-host-slash\") pod \"iptables-alerter-bwkmk\" (UID: \"3142b56b-8ab7-4799-8b7a-46420a701680\") " pod="openshift-network-operator/iptables-alerter-bwkmk"
May 20 13:50:35.334633 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333940    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-multus-socket-dir-parent\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.334633 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333963    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-etc-sysconfig\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.334633 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333943    2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 20 13:50:35.334633 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.333978    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-host-var-lib-cni-multus\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.334633 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334028    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-multus-conf-dir\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.334633 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334042    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-etc-sysconfig\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.334633 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334065    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-etc-kubernetes\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.334633 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334075    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-multus-conf-dir\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.334633 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334094    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccq69\" (UniqueName: \"kubernetes.io/projected/37b49aa0-a7ad-4499-9989-3bfc30ddf3ee-kube-api-access-ccq69\") pod \"aws-ebs-csi-driver-node-vfvkz\" (UID: \"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.334633 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334123    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-etc-kubernetes\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.335440 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334123    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-var-lib-openvswitch\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.335440 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334166    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-log-socket\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.335440 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334196    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-host-kubelet\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.335440 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334216    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tjjj4\" (UniqueName: \"kubernetes.io/projected/b985f05d-7aa5-42b7-a5d7-3b18a2769356-kube-api-access-tjjj4\") pod \"node-resolver-cs8gn\" (UID: \"b985f05d-7aa5-42b7-a5d7-3b18a2769356\") " pod="openshift-dns/node-resolver-cs8gn"
May 20 13:50:35.335440 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334231    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/efd2c3ea-0867-4025-b5c6-03b9a0d744c5-serviceca\") pod \"node-ca-dk92v\" (UID: \"efd2c3ea-0867-4025-b5c6-03b9a0d744c5\") " pod="openshift-image-registry/node-ca-dk92v"
May 20 13:50:35.335440 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334257    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2dcz\" (UniqueName: \"kubernetes.io/projected/ead02d78-f55b-4612-bb8d-057d70f6ecc4-kube-api-access-l2dcz\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.335440 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334311    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-etc-systemd\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.335440 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334343    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2mxzt\" (UniqueName: \"kubernetes.io/projected/4bdf5520-b68e-4063-b150-50ff558b61e2-kube-api-access-2mxzt\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.335440 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334369    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-cni-binary-copy\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.335440 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334389    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-etc-systemd\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.335440 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334394    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ead02d78-f55b-4612-bb8d-057d70f6ecc4-system-cni-dir\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.335440 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334426    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ead02d78-f55b-4612-bb8d-057d70f6ecc4-cni-binary-copy\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.335440 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334449    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.335440 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334490    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-etc-kubernetes\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.335440 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334511    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b985f05d-7aa5-42b7-a5d7-3b18a2769356-tmp-dir\") pod \"node-resolver-cs8gn\" (UID: \"b985f05d-7aa5-42b7-a5d7-3b18a2769356\") " pod="openshift-dns/node-resolver-cs8gn"
May 20 13:50:35.335440 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334532    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-host-run-k8s-cni-cncf-io\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.335440 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334553    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a77563fb-8a39-452d-8c02-dc39cee852cf-konnectivity-ca\") pod \"konnectivity-agent-x6mng\" (UID: \"a77563fb-8a39-452d-8c02-dc39cee852cf\") " pod="kube-system/konnectivity-agent-x6mng"
May 20 13:50:35.336045 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334564    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-etc-kubernetes\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.336045 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334574    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/37b49aa0-a7ad-4499-9989-3bfc30ddf3ee-registration-dir\") pod \"aws-ebs-csi-driver-node-vfvkz\" (UID: \"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.336045 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334593    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/37b49aa0-a7ad-4499-9989-3bfc30ddf3ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-vfvkz\" (UID: \"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.336045 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334598    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-host-run-k8s-cni-cncf-io\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.336045 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334666    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wpd62\" (UniqueName: \"kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62\") pod \"network-check-target-clw2n\" (UID: \"31842022-6a01-49b0-8b26-fb0d882cdf6f\") " pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:50:35.336045 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334687    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-host-slash\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.336045 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334702    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/efd2c3ea-0867-4025-b5c6-03b9a0d744c5-serviceca\") pod \"node-ca-dk92v\" (UID: \"efd2c3ea-0867-4025-b5c6-03b9a0d744c5\") " pod="openshift-image-registry/node-ca-dk92v"
May 20 13:50:35.336045 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334706    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-etc-modprobe-d\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.336045 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334769    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bdf5520-b68e-4063-b150-50ff558b61e2-tmp\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.336045 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334799    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xh895\" (UniqueName: \"kubernetes.io/projected/efd2c3ea-0867-4025-b5c6-03b9a0d744c5-kube-api-access-xh895\") pod \"node-ca-dk92v\" (UID: \"efd2c3ea-0867-4025-b5c6-03b9a0d744c5\") " pod="openshift-image-registry/node-ca-dk92v"
May 20 13:50:35.336045 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334825    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-os-release\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.336045 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334859    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-multus-daemon-config\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.336045 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334862    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b985f05d-7aa5-42b7-a5d7-3b18a2769356-tmp-dir\") pod \"node-resolver-cs8gn\" (UID: \"b985f05d-7aa5-42b7-a5d7-3b18a2769356\") " pod="openshift-dns/node-resolver-cs8gn"
May 20 13:50:35.336045 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334883    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5ktv2\" (UniqueName: \"kubernetes.io/projected/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-kube-api-access-5ktv2\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.336045 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334888    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-etc-modprobe-d\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.336045 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334911    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a77563fb-8a39-452d-8c02-dc39cee852cf-agent-certs\") pod \"konnectivity-agent-x6mng\" (UID: \"a77563fb-8a39-452d-8c02-dc39cee852cf\") " pod="kube-system/konnectivity-agent-x6mng"
May 20 13:50:35.336045 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334937    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/37b49aa0-a7ad-4499-9989-3bfc30ddf3ee-socket-dir\") pod \"aws-ebs-csi-driver-node-vfvkz\" (UID: \"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.336572 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334967    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-var-lib-kubelet\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.336572 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334994    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-cnibin\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.336572 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.334996    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-os-release\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.336572 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.335019    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ead02d78-f55b-4612-bb8d-057d70f6ecc4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.336572 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.335048    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-run-openvswitch\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.336572 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.335059    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-var-lib-kubelet\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.336572 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.335071    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-run-ovn\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.336572 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.335106    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.336572 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.335140    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/56992f6a-130b-435f-a75b-bf6b103ec8e1-ovn-node-metrics-cert\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.336572 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.335163    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-etc-openvswitch\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.336572 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.335189    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/56992f6a-130b-435f-a75b-bf6b103ec8e1-env-overrides\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.336572 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.335167    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-cnibin\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.336572 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.335213    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3142b56b-8ab7-4799-8b7a-46420a701680-iptables-alerter-script\") pod \"iptables-alerter-bwkmk\" (UID: \"3142b56b-8ab7-4799-8b7a-46420a701680\") " pod="openshift-network-operator/iptables-alerter-bwkmk"
May 20 13:50:35.336572 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.335225    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-cni-binary-copy\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.336572 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.335241    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-etc-sysctl-conf\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.336572 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.335269    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-host\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.336572 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.335322    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-host\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.337311 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.335334    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ead02d78-f55b-4612-bb8d-057d70f6ecc4-cnibin\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.337311 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.335356    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-multus-daemon-config\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.337311 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.335360    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ead02d78-f55b-4612-bb8d-057d70f6ecc4-os-release\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.337311 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.335386    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ead02d78-f55b-4612-bb8d-057d70f6ecc4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.337311 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.335392    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4bdf5520-b68e-4063-b150-50ff558b61e2-etc-sysctl-conf\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.337311 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.335437    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-host-run-netns\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.337311 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.335532    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/56992f6a-130b-435f-a75b-bf6b103ec8e1-ovnkube-script-lib\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.337976 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.337960    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4bdf5520-b68e-4063-b150-50ff558b61e2-etc-tuned\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.338075 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.338026    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bdf5520-b68e-4063-b150-50ff558b61e2-tmp\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.347320 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:35.347284    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:35.347320 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:35.347310    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:35.347320 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:35.347324    2570 projected.go:194] Error preparing data for projected volume kube-api-access-wpd62 for pod openshift-network-diagnostics/network-check-target-clw2n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:35.347578 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:35.347471    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62 podName:31842022-6a01-49b0-8b26-fb0d882cdf6f nodeName:}" failed. No retries permitted until 2026-05-20 13:50:35.847431487 +0000 UTC m=+3.085048124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wpd62" (UniqueName: "kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62") pod "network-check-target-clw2n" (UID: "31842022-6a01-49b0-8b26-fb0d882cdf6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:35.350171 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.350149    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjjj4\" (UniqueName: \"kubernetes.io/projected/b985f05d-7aa5-42b7-a5d7-3b18a2769356-kube-api-access-tjjj4\") pod \"node-resolver-cs8gn\" (UID: \"b985f05d-7aa5-42b7-a5d7-3b18a2769356\") " pod="openshift-dns/node-resolver-cs8gn"
May 20 13:50:35.350262 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.350166    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mxzt\" (UniqueName: \"kubernetes.io/projected/4bdf5520-b68e-4063-b150-50ff558b61e2-kube-api-access-2mxzt\") pod \"tuned-jt2w6\" (UID: \"4bdf5520-b68e-4063-b150-50ff558b61e2\") " pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.353012 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.352987    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh895\" (UniqueName: \"kubernetes.io/projected/efd2c3ea-0867-4025-b5c6-03b9a0d744c5-kube-api-access-xh895\") pod \"node-ca-dk92v\" (UID: \"efd2c3ea-0867-4025-b5c6-03b9a0d744c5\") " pod="openshift-image-registry/node-ca-dk92v"
May 20 13:50:35.353173 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.353150    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ktv2\" (UniqueName: \"kubernetes.io/projected/cc57a4cf-cc2d-43b4-a0a4-de6a545405a3-kube-api-access-5ktv2\") pod \"multus-9wtjr\" (UID: \"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3\") " pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.363162 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.363106    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal" event={"ID":"38a3d6bde081a3bde4f3c1a1f8fce848","Type":"ContainerStarted","Data":"46c2f76337e41e6cf5a07491027d88bbfc88cc806f4a40e8e0c2154a7e8be0fe"}
May 20 13:50:35.364288 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.364263    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-75.ec2.internal" event={"ID":"6959407f02e6e503e019a9457b3f87d1","Type":"ContainerStarted","Data":"8898992cf14ffb508afed109d5ebd197dc95ac0e1f2e5a2bb636d0e60bbc6599"}
May 20 13:50:35.371311 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.371284    2570 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:35.435861 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.435822    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs\") pod \"network-metrics-daemon-rhcx7\" (UID: \"6969002d-0b97-4cb6-ae5b-03d5ef9dddf8\") " pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:35.436051 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.435874    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3142b56b-8ab7-4799-8b7a-46420a701680-host-slash\") pod \"iptables-alerter-bwkmk\" (UID: \"3142b56b-8ab7-4799-8b7a-46420a701680\") " pod="openshift-network-operator/iptables-alerter-bwkmk"
May 20 13:50:35.436051 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.435904    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ccq69\" (UniqueName: \"kubernetes.io/projected/37b49aa0-a7ad-4499-9989-3bfc30ddf3ee-kube-api-access-ccq69\") pod \"aws-ebs-csi-driver-node-vfvkz\" (UID: \"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.436051 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.435921    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-var-lib-openvswitch\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.436051 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.435938    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-log-socket\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.436051 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.435959    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-host-kubelet\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.436051 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.435986    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l2dcz\" (UniqueName: \"kubernetes.io/projected/ead02d78-f55b-4612-bb8d-057d70f6ecc4-kube-api-access-l2dcz\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.436051 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:35.435988    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:35.436051 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436013    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ead02d78-f55b-4612-bb8d-057d70f6ecc4-system-cni-dir\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.436051 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436030    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-var-lib-openvswitch\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.436051 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436038    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ead02d78-f55b-4612-bb8d-057d70f6ecc4-cni-binary-copy\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.436051 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436037    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3142b56b-8ab7-4799-8b7a-46420a701680-host-slash\") pod \"iptables-alerter-bwkmk\" (UID: \"3142b56b-8ab7-4799-8b7a-46420a701680\") " pod="openshift-network-operator/iptables-alerter-bwkmk"
May 20 13:50:35.436565 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:35.436062    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs podName:6969002d-0b97-4cb6-ae5b-03d5ef9dddf8 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:35.936042605 +0000 UTC m=+3.173659247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs") pod "network-metrics-daemon-rhcx7" (UID: "6969002d-0b97-4cb6-ae5b-03d5ef9dddf8") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:35.436565 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436073    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-host-kubelet\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.436565 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436102    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.436565 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436126    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-log-socket\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.436565 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436137    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a77563fb-8a39-452d-8c02-dc39cee852cf-konnectivity-ca\") pod \"konnectivity-agent-x6mng\" (UID: \"a77563fb-8a39-452d-8c02-dc39cee852cf\") " pod="kube-system/konnectivity-agent-x6mng"
May 20 13:50:35.436565 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436164    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/37b49aa0-a7ad-4499-9989-3bfc30ddf3ee-registration-dir\") pod \"aws-ebs-csi-driver-node-vfvkz\" (UID: \"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.436565 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436190    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/37b49aa0-a7ad-4499-9989-3bfc30ddf3ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-vfvkz\" (UID: \"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.436565 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436226    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-host-slash\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.436565 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436228    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/37b49aa0-a7ad-4499-9989-3bfc30ddf3ee-registration-dir\") pod \"aws-ebs-csi-driver-node-vfvkz\" (UID: \"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.436565 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436161    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.436565 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436255    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a77563fb-8a39-452d-8c02-dc39cee852cf-agent-certs\") pod \"konnectivity-agent-x6mng\" (UID: \"a77563fb-8a39-452d-8c02-dc39cee852cf\") " pod="kube-system/konnectivity-agent-x6mng"
May 20 13:50:35.436565 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436294    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ead02d78-f55b-4612-bb8d-057d70f6ecc4-system-cni-dir\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.436565 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436413    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/37b49aa0-a7ad-4499-9989-3bfc30ddf3ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-vfvkz\" (UID: \"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.436565 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436427    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/37b49aa0-a7ad-4499-9989-3bfc30ddf3ee-socket-dir\") pod \"aws-ebs-csi-driver-node-vfvkz\" (UID: \"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.436565 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436460    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ead02d78-f55b-4612-bb8d-057d70f6ecc4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.436565 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436486    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-run-openvswitch\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.437324 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436510    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-run-ovn\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.437324 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436535    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.437324 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436555    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/37b49aa0-a7ad-4499-9989-3bfc30ddf3ee-socket-dir\") pod \"aws-ebs-csi-driver-node-vfvkz\" (UID: \"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.437324 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436559    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/56992f6a-130b-435f-a75b-bf6b103ec8e1-ovn-node-metrics-cert\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.437324 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436593    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-etc-openvswitch\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.437324 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436597    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-run-openvswitch\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.437324 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436642    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/56992f6a-130b-435f-a75b-bf6b103ec8e1-env-overrides\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.437324 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436644    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ead02d78-f55b-4612-bb8d-057d70f6ecc4-cni-binary-copy\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.437324 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436644    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a77563fb-8a39-452d-8c02-dc39cee852cf-konnectivity-ca\") pod \"konnectivity-agent-x6mng\" (UID: \"a77563fb-8a39-452d-8c02-dc39cee852cf\") " pod="kube-system/konnectivity-agent-x6mng"
May 20 13:50:35.437324 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436680    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3142b56b-8ab7-4799-8b7a-46420a701680-iptables-alerter-script\") pod \"iptables-alerter-bwkmk\" (UID: \"3142b56b-8ab7-4799-8b7a-46420a701680\") " pod="openshift-network-operator/iptables-alerter-bwkmk"
May 20 13:50:35.437324 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436701    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-run-ovn\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.437324 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436714    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ead02d78-f55b-4612-bb8d-057d70f6ecc4-cnibin\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.437324 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436738    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-etc-openvswitch\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.437324 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436772    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ead02d78-f55b-4612-bb8d-057d70f6ecc4-cnibin\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.437324 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436775    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ead02d78-f55b-4612-bb8d-057d70f6ecc4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.437324 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436458    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-host-slash\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.437324 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436820    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.437973 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436838    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ead02d78-f55b-4612-bb8d-057d70f6ecc4-os-release\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.437973 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436739    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ead02d78-f55b-4612-bb8d-057d70f6ecc4-os-release\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.437973 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436874    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ead02d78-f55b-4612-bb8d-057d70f6ecc4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.437973 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436899    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-host-run-netns\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.437973 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.436924    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/56992f6a-130b-435f-a75b-bf6b103ec8e1-ovnkube-script-lib\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.437973 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437164    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3142b56b-8ab7-4799-8b7a-46420a701680-iptables-alerter-script\") pod \"iptables-alerter-bwkmk\" (UID: \"3142b56b-8ab7-4799-8b7a-46420a701680\") " pod="openshift-network-operator/iptables-alerter-bwkmk"
May 20 13:50:35.437973 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437244    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mrl92\" (UniqueName: \"kubernetes.io/projected/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-kube-api-access-mrl92\") pod \"network-metrics-daemon-rhcx7\" (UID: \"6969002d-0b97-4cb6-ae5b-03d5ef9dddf8\") " pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:35.437973 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437262    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/56992f6a-130b-435f-a75b-bf6b103ec8e1-env-overrides\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.437973 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437278    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-node-log\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.437973 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437309    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-host-cni-netd\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.437973 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437335    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-954t2\" (UniqueName: \"kubernetes.io/projected/3142b56b-8ab7-4799-8b7a-46420a701680-kube-api-access-954t2\") pod \"iptables-alerter-bwkmk\" (UID: \"3142b56b-8ab7-4799-8b7a-46420a701680\") " pod="openshift-network-operator/iptables-alerter-bwkmk"
May 20 13:50:35.437973 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437347    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-node-log\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.437973 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437361    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-host-cni-bin\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.437973 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437386    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8l2gz\" (UniqueName: \"kubernetes.io/projected/56992f6a-130b-435f-a75b-bf6b103ec8e1-kube-api-access-8l2gz\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.437973 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437399    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-host-cni-netd\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.437973 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437407    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ead02d78-f55b-4612-bb8d-057d70f6ecc4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.437973 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437420    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-host-cni-bin\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.438554 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437386    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/56992f6a-130b-435f-a75b-bf6b103ec8e1-ovnkube-script-lib\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.438554 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437413    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/37b49aa0-a7ad-4499-9989-3bfc30ddf3ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vfvkz\" (UID: \"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.438554 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437470    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/56992f6a-130b-435f-a75b-bf6b103ec8e1-ovnkube-config\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.438554 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437476    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/37b49aa0-a7ad-4499-9989-3bfc30ddf3ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vfvkz\" (UID: \"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.438554 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437499    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/37b49aa0-a7ad-4499-9989-3bfc30ddf3ee-device-dir\") pod \"aws-ebs-csi-driver-node-vfvkz\" (UID: \"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.438554 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437528    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/37b49aa0-a7ad-4499-9989-3bfc30ddf3ee-sys-fs\") pod \"aws-ebs-csi-driver-node-vfvkz\" (UID: \"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.438554 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437554    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ead02d78-f55b-4612-bb8d-057d70f6ecc4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.438554 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437577    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/37b49aa0-a7ad-4499-9989-3bfc30ddf3ee-device-dir\") pod \"aws-ebs-csi-driver-node-vfvkz\" (UID: \"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.438554 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437578    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-systemd-units\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.438554 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437590    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/37b49aa0-a7ad-4499-9989-3bfc30ddf3ee-sys-fs\") pod \"aws-ebs-csi-driver-node-vfvkz\" (UID: \"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.438554 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437632    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-run-systemd\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.438554 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437635    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-systemd-units\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.438554 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437698    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-run-systemd\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.438554 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437735    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56992f6a-130b-435f-a75b-bf6b103ec8e1-host-run-netns\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.438554 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.437987    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ead02d78-f55b-4612-bb8d-057d70f6ecc4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.438554 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.438436    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/56992f6a-130b-435f-a75b-bf6b103ec8e1-ovnkube-config\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.439279 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.439258    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/56992f6a-130b-435f-a75b-bf6b103ec8e1-ovn-node-metrics-cert\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.439426 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.439405    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a77563fb-8a39-452d-8c02-dc39cee852cf-agent-certs\") pod \"konnectivity-agent-x6mng\" (UID: \"a77563fb-8a39-452d-8c02-dc39cee852cf\") " pod="kube-system/konnectivity-agent-x6mng"
May 20 13:50:35.446506 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.446483    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2dcz\" (UniqueName: \"kubernetes.io/projected/ead02d78-f55b-4612-bb8d-057d70f6ecc4-kube-api-access-l2dcz\") pod \"multus-additional-cni-plugins-zgj4t\" (UID: \"ead02d78-f55b-4612-bb8d-057d70f6ecc4\") " pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.448345 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.448324    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-954t2\" (UniqueName: \"kubernetes.io/projected/3142b56b-8ab7-4799-8b7a-46420a701680-kube-api-access-954t2\") pod \"iptables-alerter-bwkmk\" (UID: \"3142b56b-8ab7-4799-8b7a-46420a701680\") " pod="openshift-network-operator/iptables-alerter-bwkmk"
May 20 13:50:35.448459 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.448425    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l2gz\" (UniqueName: \"kubernetes.io/projected/56992f6a-130b-435f-a75b-bf6b103ec8e1-kube-api-access-8l2gz\") pod \"ovnkube-node-kfhg5\" (UID: \"56992f6a-130b-435f-a75b-bf6b103ec8e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.449188 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.449169    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccq69\" (UniqueName: \"kubernetes.io/projected/37b49aa0-a7ad-4499-9989-3bfc30ddf3ee-kube-api-access-ccq69\") pod \"aws-ebs-csi-driver-node-vfvkz\" (UID: \"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.451651 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.451586    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrl92\" (UniqueName: \"kubernetes.io/projected/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-kube-api-access-mrl92\") pod \"network-metrics-daemon-rhcx7\" (UID: \"6969002d-0b97-4cb6-ae5b-03d5ef9dddf8\") " pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:35.521414 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.521379    2570 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:35.525607 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.525582    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jt2w6"
May 20 13:50:35.537328 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.537301    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cs8gn"
May 20 13:50:35.545829 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.545807    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dk92v"
May 20 13:50:35.551403 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.551384    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9wtjr"
May 20 13:50:35.559134 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.559112    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:35.566859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.566839    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x6mng"
May 20 13:50:35.580523 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.580488    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz"
May 20 13:50:35.587244 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.587221    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zgj4t"
May 20 13:50:35.594950 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.594930    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bwkmk"
May 20 13:50:35.940771 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.940740    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs\") pod \"network-metrics-daemon-rhcx7\" (UID: \"6969002d-0b97-4cb6-ae5b-03d5ef9dddf8\") " pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:35.940953 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:35.940796    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wpd62\" (UniqueName: \"kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62\") pod \"network-check-target-clw2n\" (UID: \"31842022-6a01-49b0-8b26-fb0d882cdf6f\") " pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:50:35.940953 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:35.940914    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:35.940953 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:35.940944    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:35.941103 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:35.940964    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:35.941103 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:35.940976    2570 projected.go:194] Error preparing data for projected volume kube-api-access-wpd62 for pod openshift-network-diagnostics/network-check-target-clw2n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:35.941103 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:35.940988    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs podName:6969002d-0b97-4cb6-ae5b-03d5ef9dddf8 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:36.940967221 +0000 UTC m=+4.178583862 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs") pod "network-metrics-daemon-rhcx7" (UID: "6969002d-0b97-4cb6-ae5b-03d5ef9dddf8") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:35.941103 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:35.941029    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62 podName:31842022-6a01-49b0-8b26-fb0d882cdf6f nodeName:}" failed. No retries permitted until 2026-05-20 13:50:36.941010963 +0000 UTC m=+4.178627602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wpd62" (UniqueName: "kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62") pod "network-check-target-clw2n" (UID: "31842022-6a01-49b0-8b26-fb0d882cdf6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:36.073665 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:36.073635    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead02d78_f55b_4612_bb8d_057d70f6ecc4.slice/crio-f987be9d157599f0dc21a655b2a700cc6ece869e1508b4189de285bd41578204 WatchSource:0}: Error finding container f987be9d157599f0dc21a655b2a700cc6ece869e1508b4189de285bd41578204: Status 404 returned error can't find the container with id f987be9d157599f0dc21a655b2a700cc6ece869e1508b4189de285bd41578204
May 20 13:50:36.076305 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:36.076273    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bdf5520_b68e_4063_b150_50ff558b61e2.slice/crio-fd2e73150ca60786be11266a2806874d526f37d38edb509ec90eea821e165ee1 WatchSource:0}: Error finding container fd2e73150ca60786be11266a2806874d526f37d38edb509ec90eea821e165ee1: Status 404 returned error can't find the container with id fd2e73150ca60786be11266a2806874d526f37d38edb509ec90eea821e165ee1
May 20 13:50:36.077362 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:36.077333    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3142b56b_8ab7_4799_8b7a_46420a701680.slice/crio-8ee306abe21f1cf2972c3eaaebe0fde209ccc815800ed9e85cf0bc7971921309 WatchSource:0}: Error finding container 8ee306abe21f1cf2972c3eaaebe0fde209ccc815800ed9e85cf0bc7971921309: Status 404 returned error can't find the container with id 8ee306abe21f1cf2972c3eaaebe0fde209ccc815800ed9e85cf0bc7971921309
May 20 13:50:36.080778 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:36.080757    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc57a4cf_cc2d_43b4_a0a4_de6a545405a3.slice/crio-1870f135af6cbe28ceb98f97a17d93d83a2601da17233dd7cbb2bd328df17fd0 WatchSource:0}: Error finding container 1870f135af6cbe28ceb98f97a17d93d83a2601da17233dd7cbb2bd328df17fd0: Status 404 returned error can't find the container with id 1870f135af6cbe28ceb98f97a17d93d83a2601da17233dd7cbb2bd328df17fd0
May 20 13:50:36.081996 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:36.081931    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b49aa0_a7ad_4499_9989_3bfc30ddf3ee.slice/crio-365fd67ece065457db0c274b34aaabcd6a0d38904dc6fe0c08b52d3249ebb7e2 WatchSource:0}: Error finding container 365fd67ece065457db0c274b34aaabcd6a0d38904dc6fe0c08b52d3249ebb7e2: Status 404 returned error can't find the container with id 365fd67ece065457db0c274b34aaabcd6a0d38904dc6fe0c08b52d3249ebb7e2
May 20 13:50:36.082398 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:36.082375    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56992f6a_130b_435f_a75b_bf6b103ec8e1.slice/crio-82fa79b973ada044f6528749f74f0f916c5f30ab5e96836592ffea2493139c15 WatchSource:0}: Error finding container 82fa79b973ada044f6528749f74f0f916c5f30ab5e96836592ffea2493139c15: Status 404 returned error can't find the container with id 82fa79b973ada044f6528749f74f0f916c5f30ab5e96836592ffea2493139c15
May 20 13:50:36.084164 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:36.084060    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77563fb_8a39_452d_8c02_dc39cee852cf.slice/crio-a2e5cc819a038f82c79022510ccf8779c86a072fcb24ade1471da2b9fc7a8b21 WatchSource:0}: Error finding container a2e5cc819a038f82c79022510ccf8779c86a072fcb24ade1471da2b9fc7a8b21: Status 404 returned error can't find the container with id a2e5cc819a038f82c79022510ccf8779c86a072fcb24ade1471da2b9fc7a8b21
May 20 13:50:36.084597 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:36.084571    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb985f05d_7aa5_42b7_a5d7_3b18a2769356.slice/crio-7146b6c207031b0d6817cd3a11c6e0c683426a7cfc74aaf7a654653f83faa5ce WatchSource:0}: Error finding container 7146b6c207031b0d6817cd3a11c6e0c683426a7cfc74aaf7a654653f83faa5ce: Status 404 returned error can't find the container with id 7146b6c207031b0d6817cd3a11c6e0c683426a7cfc74aaf7a654653f83faa5ce
May 20 13:50:36.085666 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:50:36.085346    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd2c3ea_0867_4025_b5c6_03b9a0d744c5.slice/crio-7c18a1c1fe072ffb4662ab6b5d421e35d97bf8249d2bf9ca9bc4f76601010134 WatchSource:0}: Error finding container 7c18a1c1fe072ffb4662ab6b5d421e35d97bf8249d2bf9ca9bc4f76601010134: Status 404 returned error can't find the container with id 7c18a1c1fe072ffb4662ab6b5d421e35d97bf8249d2bf9ca9bc4f76601010134
May 20 13:50:36.292778 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:36.292542    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 13:45:34 +0000 UTC" deadline="2028-03-01 19:40:50.101405054 +0000 UTC"
May 20 13:50:36.292778 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:36.292778    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15629h50m13.808633696s"
May 20 13:50:36.367698 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:36.367661    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-75.ec2.internal" event={"ID":"6959407f02e6e503e019a9457b3f87d1","Type":"ContainerStarted","Data":"74f0e52af185af163baa04afdee73a7a4f67ecbe9b0cbbf05b70baffbdb3cb83"}
May 20 13:50:36.368916 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:36.368885    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dk92v" event={"ID":"efd2c3ea-0867-4025-b5c6-03b9a0d744c5","Type":"ContainerStarted","Data":"7c18a1c1fe072ffb4662ab6b5d421e35d97bf8249d2bf9ca9bc4f76601010134"}
May 20 13:50:36.369756 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:36.369736    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x6mng" event={"ID":"a77563fb-8a39-452d-8c02-dc39cee852cf","Type":"ContainerStarted","Data":"a2e5cc819a038f82c79022510ccf8779c86a072fcb24ade1471da2b9fc7a8b21"}
May 20 13:50:36.370544 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:36.370521    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz" event={"ID":"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee","Type":"ContainerStarted","Data":"365fd67ece065457db0c274b34aaabcd6a0d38904dc6fe0c08b52d3249ebb7e2"}
May 20 13:50:36.371681 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:36.371470    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9wtjr" event={"ID":"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3","Type":"ContainerStarted","Data":"1870f135af6cbe28ceb98f97a17d93d83a2601da17233dd7cbb2bd328df17fd0"}
May 20 13:50:36.372498 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:36.372479    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bwkmk" event={"ID":"3142b56b-8ab7-4799-8b7a-46420a701680","Type":"ContainerStarted","Data":"8ee306abe21f1cf2972c3eaaebe0fde209ccc815800ed9e85cf0bc7971921309"}
May 20 13:50:36.373685 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:36.373663    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal" event={"ID":"38a3d6bde081a3bde4f3c1a1f8fce848","Type":"ContainerStarted","Data":"958129026bf92c96e4962a6241e5707c7b241a8d7cd9c47cbaebaa82f1ec68e4"}
May 20 13:50:36.374669 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:36.374642    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cs8gn" event={"ID":"b985f05d-7aa5-42b7-a5d7-3b18a2769356","Type":"ContainerStarted","Data":"7146b6c207031b0d6817cd3a11c6e0c683426a7cfc74aaf7a654653f83faa5ce"}
May 20 13:50:36.375766 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:36.375741    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5" event={"ID":"56992f6a-130b-435f-a75b-bf6b103ec8e1","Type":"ContainerStarted","Data":"82fa79b973ada044f6528749f74f0f916c5f30ab5e96836592ffea2493139c15"}
May 20 13:50:36.376776 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:36.376754    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jt2w6" event={"ID":"4bdf5520-b68e-4063-b150-50ff558b61e2","Type":"ContainerStarted","Data":"fd2e73150ca60786be11266a2806874d526f37d38edb509ec90eea821e165ee1"}
May 20 13:50:36.377977 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:36.377956    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zgj4t" event={"ID":"ead02d78-f55b-4612-bb8d-057d70f6ecc4","Type":"ContainerStarted","Data":"f987be9d157599f0dc21a655b2a700cc6ece869e1508b4189de285bd41578204"}
May 20 13:50:36.394144 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:36.394039    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-75.ec2.internal" podStartSLOduration=2.394024352 podStartE2EDuration="2.394024352s" podCreationTimestamp="2026-05-20 13:50:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:50:36.393604671 +0000 UTC m=+3.631221319" watchObservedRunningTime="2026-05-20 13:50:36.394024352 +0000 UTC m=+3.631640997"
May 20 13:50:36.949417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:36.949378    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs\") pod \"network-metrics-daemon-rhcx7\" (UID: \"6969002d-0b97-4cb6-ae5b-03d5ef9dddf8\") " pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:36.949637 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:36.949447    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wpd62\" (UniqueName: \"kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62\") pod \"network-check-target-clw2n\" (UID: \"31842022-6a01-49b0-8b26-fb0d882cdf6f\") " pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:50:36.949637 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:36.949628    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:36.949753 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:36.949651    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:36.949753 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:36.949667    2570 projected.go:194] Error preparing data for projected volume kube-api-access-wpd62 for pod openshift-network-diagnostics/network-check-target-clw2n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:36.949753 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:36.949728    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62 podName:31842022-6a01-49b0-8b26-fb0d882cdf6f nodeName:}" failed. No retries permitted until 2026-05-20 13:50:38.949709173 +0000 UTC m=+6.187325801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wpd62" (UniqueName: "kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62") pod "network-check-target-clw2n" (UID: "31842022-6a01-49b0-8b26-fb0d882cdf6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:36.950146 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:36.950118    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:36.950208 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:36.950172    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs podName:6969002d-0b97-4cb6-ae5b-03d5ef9dddf8 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:38.950155421 +0000 UTC m=+6.187772048 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs") pod "network-metrics-daemon-rhcx7" (UID: "6969002d-0b97-4cb6-ae5b-03d5ef9dddf8") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:37.362041 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:37.362006    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:37.362498 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:37.362193    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhcx7" podUID="6969002d-0b97-4cb6-ae5b-03d5ef9dddf8"
May 20 13:50:37.362498 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:37.362291    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:50:37.362498 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:37.362372    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clw2n" podUID="31842022-6a01-49b0-8b26-fb0d882cdf6f"
May 20 13:50:37.395411 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:37.395368    2570 generic.go:358] "Generic (PLEG): container finished" podID="38a3d6bde081a3bde4f3c1a1f8fce848" containerID="958129026bf92c96e4962a6241e5707c7b241a8d7cd9c47cbaebaa82f1ec68e4" exitCode=0
May 20 13:50:37.395990 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:37.395963    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal" event={"ID":"38a3d6bde081a3bde4f3c1a1f8fce848","Type":"ContainerDied","Data":"958129026bf92c96e4962a6241e5707c7b241a8d7cd9c47cbaebaa82f1ec68e4"}
May 20 13:50:38.407465 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:38.406830    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal" event={"ID":"38a3d6bde081a3bde4f3c1a1f8fce848","Type":"ContainerStarted","Data":"584315c94cfaf23448e77d0962460df36d5d97b44aef9de79286d6327c90d6ec"}
May 20 13:50:38.428899 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:38.428325    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal" podStartSLOduration=4.428306515 podStartE2EDuration="4.428306515s" podCreationTimestamp="2026-05-20 13:50:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:50:38.427791875 +0000 UTC m=+5.665408525" watchObservedRunningTime="2026-05-20 13:50:38.428306515 +0000 UTC m=+5.665923165"
May 20 13:50:38.969530 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:38.969486    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wpd62\" (UniqueName: \"kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62\") pod \"network-check-target-clw2n\" (UID: \"31842022-6a01-49b0-8b26-fb0d882cdf6f\") " pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:50:38.969806 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:38.969572    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs\") pod \"network-metrics-daemon-rhcx7\" (UID: \"6969002d-0b97-4cb6-ae5b-03d5ef9dddf8\") " pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:38.969806 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:38.969699    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:38.969806 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:38.969748    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs podName:6969002d-0b97-4cb6-ae5b-03d5ef9dddf8 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:42.96973518 +0000 UTC m=+10.207351805 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs") pod "network-metrics-daemon-rhcx7" (UID: "6969002d-0b97-4cb6-ae5b-03d5ef9dddf8") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:38.970092 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:38.970075    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:38.970156 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:38.970096    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:38.970156 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:38.970105    2570 projected.go:194] Error preparing data for projected volume kube-api-access-wpd62 for pod openshift-network-diagnostics/network-check-target-clw2n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:38.970156 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:38.970137    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62 podName:31842022-6a01-49b0-8b26-fb0d882cdf6f nodeName:}" failed. No retries permitted until 2026-05-20 13:50:42.97012638 +0000 UTC m=+10.207743005 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wpd62" (UniqueName: "kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62") pod "network-check-target-clw2n" (UID: "31842022-6a01-49b0-8b26-fb0d882cdf6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:39.359834 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:39.359800    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:39.359834 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:39.359835    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:50:39.360078 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:39.359955    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhcx7" podUID="6969002d-0b97-4cb6-ae5b-03d5ef9dddf8"
May 20 13:50:39.360164 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:39.360092    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clw2n" podUID="31842022-6a01-49b0-8b26-fb0d882cdf6f"
May 20 13:50:41.362377 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:41.362345    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:41.362848 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:41.362485    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhcx7" podUID="6969002d-0b97-4cb6-ae5b-03d5ef9dddf8"
May 20 13:50:41.362907 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:41.362890    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:50:41.363008 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:41.362981    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clw2n" podUID="31842022-6a01-49b0-8b26-fb0d882cdf6f"
May 20 13:50:43.007000 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:43.006958    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wpd62\" (UniqueName: \"kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62\") pod \"network-check-target-clw2n\" (UID: \"31842022-6a01-49b0-8b26-fb0d882cdf6f\") " pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:50:43.007466 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:43.007046    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs\") pod \"network-metrics-daemon-rhcx7\" (UID: \"6969002d-0b97-4cb6-ae5b-03d5ef9dddf8\") " pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:43.007466 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:43.007229    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:43.007466 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:43.007305    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs podName:6969002d-0b97-4cb6-ae5b-03d5ef9dddf8 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:51.007284197 +0000 UTC m=+18.244900834 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs") pod "network-metrics-daemon-rhcx7" (UID: "6969002d-0b97-4cb6-ae5b-03d5ef9dddf8") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:43.007466 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:43.007415    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:43.007466 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:43.007429    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:43.007466 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:43.007456    2570 projected.go:194] Error preparing data for projected volume kube-api-access-wpd62 for pod openshift-network-diagnostics/network-check-target-clw2n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:43.007829 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:43.007499    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62 podName:31842022-6a01-49b0-8b26-fb0d882cdf6f nodeName:}" failed. No retries permitted until 2026-05-20 13:50:51.007485091 +0000 UTC m=+18.245101729 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wpd62" (UniqueName: "kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62") pod "network-check-target-clw2n" (UID: "31842022-6a01-49b0-8b26-fb0d882cdf6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:43.363292 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:43.362733    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:50:43.363292 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:43.362774    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:43.363292 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:43.362847    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clw2n" podUID="31842022-6a01-49b0-8b26-fb0d882cdf6f"
May 20 13:50:43.363292 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:43.363243    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhcx7" podUID="6969002d-0b97-4cb6-ae5b-03d5ef9dddf8"
May 20 13:50:45.359857 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:45.359819    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:45.360212 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:45.359970    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhcx7" podUID="6969002d-0b97-4cb6-ae5b-03d5ef9dddf8"
May 20 13:50:45.360339 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:45.359819    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:50:45.360475 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:45.360422    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clw2n" podUID="31842022-6a01-49b0-8b26-fb0d882cdf6f"
May 20 13:50:47.358945 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:47.358907    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:47.359374 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:47.358908    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:50:47.359374 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:47.359057    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhcx7" podUID="6969002d-0b97-4cb6-ae5b-03d5ef9dddf8"
May 20 13:50:47.359374 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:47.359093    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clw2n" podUID="31842022-6a01-49b0-8b26-fb0d882cdf6f"
May 20 13:50:49.359044 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:49.358953    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:49.359044 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:49.358999    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:50:49.359579 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:49.359111    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhcx7" podUID="6969002d-0b97-4cb6-ae5b-03d5ef9dddf8"
May 20 13:50:49.359579 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:49.359238    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clw2n" podUID="31842022-6a01-49b0-8b26-fb0d882cdf6f"
May 20 13:50:51.065597 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:51.065552    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wpd62\" (UniqueName: \"kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62\") pod \"network-check-target-clw2n\" (UID: \"31842022-6a01-49b0-8b26-fb0d882cdf6f\") " pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:50:51.066069 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:51.065657    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs\") pod \"network-metrics-daemon-rhcx7\" (UID: \"6969002d-0b97-4cb6-ae5b-03d5ef9dddf8\") " pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:51.066069 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:51.065758    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:51.066069 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:51.065790    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:51.066069 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:51.065763    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:51.066069 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:51.065803    2570 projected.go:194] Error preparing data for projected volume kube-api-access-wpd62 for pod openshift-network-diagnostics/network-check-target-clw2n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:51.066069 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:51.065860    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs podName:6969002d-0b97-4cb6-ae5b-03d5ef9dddf8 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:07.065840592 +0000 UTC m=+34.303457220 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs") pod "network-metrics-daemon-rhcx7" (UID: "6969002d-0b97-4cb6-ae5b-03d5ef9dddf8") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:51.066069 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:51.065877    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62 podName:31842022-6a01-49b0-8b26-fb0d882cdf6f nodeName:}" failed. No retries permitted until 2026-05-20 13:51:07.065868583 +0000 UTC m=+34.303485214 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wpd62" (UniqueName: "kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62") pod "network-check-target-clw2n" (UID: "31842022-6a01-49b0-8b26-fb0d882cdf6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:51.359351 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:51.359260    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:51.359530 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:51.359406    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhcx7" podUID="6969002d-0b97-4cb6-ae5b-03d5ef9dddf8"
May 20 13:50:51.359530 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:51.359412    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:50:51.359530 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:51.359499    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clw2n" podUID="31842022-6a01-49b0-8b26-fb0d882cdf6f"
May 20 13:50:53.360812 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:53.360773    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:53.361279 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:53.360974    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhcx7" podUID="6969002d-0b97-4cb6-ae5b-03d5ef9dddf8"
May 20 13:50:53.361279 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:53.361020    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:50:53.361279 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:53.361116    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clw2n" podUID="31842022-6a01-49b0-8b26-fb0d882cdf6f"
May 20 13:50:55.360075 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.359872    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:50:55.360831 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.359921    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:55.360831 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:55.360171    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clw2n" podUID="31842022-6a01-49b0-8b26-fb0d882cdf6f"
May 20 13:50:55.360831 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:55.360222    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhcx7" podUID="6969002d-0b97-4cb6-ae5b-03d5ef9dddf8"
May 20 13:50:55.435681 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.435646    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dk92v" event={"ID":"efd2c3ea-0867-4025-b5c6-03b9a0d744c5","Type":"ContainerStarted","Data":"76f70c2df306e967a931d0c6949eda1532db613bc8532490ae477e05e3395be4"}
May 20 13:50:55.436956 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.436927    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x6mng" event={"ID":"a77563fb-8a39-452d-8c02-dc39cee852cf","Type":"ContainerStarted","Data":"f271d5d94f7fef352164ea4154caf44a2a89913aa95c70d09621f21bb03677d0"}
May 20 13:50:55.438195 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.438174    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz" event={"ID":"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee","Type":"ContainerStarted","Data":"f7c5ec6a6d02d91db5eba54218ada6d67d4db082ec6509dc305269b647bcb122"}
May 20 13:50:55.439431 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.439408    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9wtjr" event={"ID":"cc57a4cf-cc2d-43b4-a0a4-de6a545405a3","Type":"ContainerStarted","Data":"4e44e2b3690c70b5d08a27843fac09a17db06f7e2189b2f39305bbe9b604d0fb"}
May 20 13:50:55.442753 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.442727    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cs8gn" event={"ID":"b985f05d-7aa5-42b7-a5d7-3b18a2769356","Type":"ContainerStarted","Data":"e8fb6c05c4215af5a1860ac4eca90407c81934bb4040f77f99a3e56c981dd8bd"}
May 20 13:50:55.445003 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.444986    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 13:50:55.445260 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.445243    2570 generic.go:358] "Generic (PLEG): container finished" podID="56992f6a-130b-435f-a75b-bf6b103ec8e1" containerID="f9375f77ce26401a984567c70bb5b91ea132dd9b92f7b8387eb1dcec4a873ef6" exitCode=1
May 20 13:50:55.445328 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.445291    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5" event={"ID":"56992f6a-130b-435f-a75b-bf6b103ec8e1","Type":"ContainerStarted","Data":"13b4382ef829851c8873a137dce0d59f31f1b9480dce5394fb75fb9342975600"}
May 20 13:50:55.445328 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.445304    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5" event={"ID":"56992f6a-130b-435f-a75b-bf6b103ec8e1","Type":"ContainerStarted","Data":"0887fcc5473126ec9f9a51c29452384aa90c26202408340f70748f2caefabaff"}
May 20 13:50:55.445328 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.445313    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5" event={"ID":"56992f6a-130b-435f-a75b-bf6b103ec8e1","Type":"ContainerStarted","Data":"5c339355ae09cd953167f8d21449099cc9a914a9b310710aed50dd74fcc637ee"}
May 20 13:50:55.445328 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.445321    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5" event={"ID":"56992f6a-130b-435f-a75b-bf6b103ec8e1","Type":"ContainerStarted","Data":"03d5c42afd6cee41d7877e077a411e3f8fea578c9d20bf37c6a406eb87073ca0"}
May 20 13:50:55.445328 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.445329    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5" event={"ID":"56992f6a-130b-435f-a75b-bf6b103ec8e1","Type":"ContainerDied","Data":"f9375f77ce26401a984567c70bb5b91ea132dd9b92f7b8387eb1dcec4a873ef6"}
May 20 13:50:55.445535 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.445338    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5" event={"ID":"56992f6a-130b-435f-a75b-bf6b103ec8e1","Type":"ContainerStarted","Data":"95514ce6c3103124f620467215ee6a829065c851f4705c7ef58791cf2f56c467"}
May 20 13:50:55.446347 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.446319    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jt2w6" event={"ID":"4bdf5520-b68e-4063-b150-50ff558b61e2","Type":"ContainerStarted","Data":"76009fef415c4a179ef0f690a2ff1ef684d087004e9e1e2272a17cee33afeed0"}
May 20 13:50:55.447417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.447398    2570 generic.go:358] "Generic (PLEG): container finished" podID="ead02d78-f55b-4612-bb8d-057d70f6ecc4" containerID="1799642a914f1c80cb334fb0c4d22d0ebc2c9a371212ff05a28d21447bcda658" exitCode=0
May 20 13:50:55.447494 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.447421    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zgj4t" event={"ID":"ead02d78-f55b-4612-bb8d-057d70f6ecc4","Type":"ContainerDied","Data":"1799642a914f1c80cb334fb0c4d22d0ebc2c9a371212ff05a28d21447bcda658"}
May 20 13:50:55.475394 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.475348    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dk92v" podStartSLOduration=8.995139319 podStartE2EDuration="22.47533141s" podCreationTimestamp="2026-05-20 13:50:33 +0000 UTC" firstStartedPulling="2026-05-20 13:50:36.087952872 +0000 UTC m=+3.325569503" lastFinishedPulling="2026-05-20 13:50:49.568144954 +0000 UTC m=+16.805761594" observedRunningTime="2026-05-20 13:50:55.453788334 +0000 UTC m=+22.691404980" watchObservedRunningTime="2026-05-20 13:50:55.47533141 +0000 UTC m=+22.712948058"
May 20 13:50:55.475476 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.475453    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-9wtjr" podStartSLOduration=4.184138899 podStartE2EDuration="22.475447757s" podCreationTimestamp="2026-05-20 13:50:33 +0000 UTC" firstStartedPulling="2026-05-20 13:50:36.082451536 +0000 UTC m=+3.320068161" lastFinishedPulling="2026-05-20 13:50:54.373760389 +0000 UTC m=+21.611377019" observedRunningTime="2026-05-20 13:50:55.47324497 +0000 UTC m=+22.710861615" watchObservedRunningTime="2026-05-20 13:50:55.475447757 +0000 UTC m=+22.713064453"
May 20 13:50:55.476122 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.476093    2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 20 13:50:55.498283 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.498238    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x6mng" podStartSLOduration=4.249439616 podStartE2EDuration="22.498223739s" podCreationTimestamp="2026-05-20 13:50:33 +0000 UTC" firstStartedPulling="2026-05-20 13:50:36.086736788 +0000 UTC m=+3.324353412" lastFinishedPulling="2026-05-20 13:50:54.335520907 +0000 UTC m=+21.573137535" observedRunningTime="2026-05-20 13:50:55.497876906 +0000 UTC m=+22.735493562" watchObservedRunningTime="2026-05-20 13:50:55.498223739 +0000 UTC m=+22.735840399"
May 20 13:50:55.517584 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.517545    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-cs8gn" podStartSLOduration=4.268630094 podStartE2EDuration="22.517527846s" podCreationTimestamp="2026-05-20 13:50:33 +0000 UTC" firstStartedPulling="2026-05-20 13:50:36.086636439 +0000 UTC m=+3.324253070" lastFinishedPulling="2026-05-20 13:50:54.33553418 +0000 UTC m=+21.573150822" observedRunningTime="2026-05-20 13:50:55.517515524 +0000 UTC m=+22.755132181" watchObservedRunningTime="2026-05-20 13:50:55.517527846 +0000 UTC m=+22.755144492"
May 20 13:50:55.565835 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:55.565788    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jt2w6" podStartSLOduration=4.289605283 podStartE2EDuration="22.56577309s" podCreationTimestamp="2026-05-20 13:50:33 +0000 UTC" firstStartedPulling="2026-05-20 13:50:36.079876546 +0000 UTC m=+3.317493185" lastFinishedPulling="2026-05-20 13:50:54.356044354 +0000 UTC m=+21.593660992" observedRunningTime="2026-05-20 13:50:55.564996975 +0000 UTC m=+22.802613633" watchObservedRunningTime="2026-05-20 13:50:55.56577309 +0000 UTC m=+22.803389738"
May 20 13:50:56.067978 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:56.067942    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x6mng"
May 20 13:50:56.068876 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:56.068857    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x6mng"
May 20 13:50:56.301826 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:56.301662    2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-20T13:50:55.476105746Z","UUID":"f3c2b059-2638-4a36-9a27-8c588a7b2092","Handler":null,"Name":"","Endpoint":""}
May 20 13:50:56.303591 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:56.303562    2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 20 13:50:56.303762 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:56.303599    2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 20 13:50:56.452272 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:56.452233    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz" event={"ID":"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee","Type":"ContainerStarted","Data":"93a71022e92b63bfc8f0100a9f3c3161433b95428077ab42c2404789fa47d3b6"}
May 20 13:50:56.452803 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:56.452280    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz" event={"ID":"37b49aa0-a7ad-4499-9989-3bfc30ddf3ee","Type":"ContainerStarted","Data":"e8d4e8d41655e5fafdf546d3cb9808c41fea2d7da41c19498d38fff2d2d9b8f2"}
May 20 13:50:56.454347 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:56.454312    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bwkmk" event={"ID":"3142b56b-8ab7-4799-8b7a-46420a701680","Type":"ContainerStarted","Data":"8020df79efdc524a339cd4812849d724521386710c477949f7aafdf21a8bb3ae"}
May 20 13:50:56.454474 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:56.454367    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x6mng"
May 20 13:50:56.454885 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:56.454860    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x6mng"
May 20 13:50:56.505673 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:56.505597    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfvkz" podStartSLOduration=3.383874342 podStartE2EDuration="23.505577974s" podCreationTimestamp="2026-05-20 13:50:33 +0000 UTC" firstStartedPulling="2026-05-20 13:50:36.083903485 +0000 UTC m=+3.321520111" lastFinishedPulling="2026-05-20 13:50:56.2056071 +0000 UTC m=+23.443223743" observedRunningTime="2026-05-20 13:50:56.481826519 +0000 UTC m=+23.719443179" watchObservedRunningTime="2026-05-20 13:50:56.505577974 +0000 UTC m=+23.743194603"
May 20 13:50:56.529261 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:56.529205    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bwkmk" podStartSLOduration=5.296634974 podStartE2EDuration="23.529185771s" podCreationTimestamp="2026-05-20 13:50:33 +0000 UTC" firstStartedPulling="2026-05-20 13:50:36.080811324 +0000 UTC m=+3.318427955" lastFinishedPulling="2026-05-20 13:50:54.313362114 +0000 UTC m=+21.550978752" observedRunningTime="2026-05-20 13:50:56.528863256 +0000 UTC m=+23.766479927" watchObservedRunningTime="2026-05-20 13:50:56.529185771 +0000 UTC m=+23.766802417"
May 20 13:50:57.359747 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:57.359455    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:57.359747 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:57.359484    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:50:57.359957 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:57.359833    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhcx7" podUID="6969002d-0b97-4cb6-ae5b-03d5ef9dddf8"
May 20 13:50:57.359957 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:57.359939    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clw2n" podUID="31842022-6a01-49b0-8b26-fb0d882cdf6f"
May 20 13:50:57.460064 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:57.460029    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 13:50:57.460537 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:57.460500    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5" event={"ID":"56992f6a-130b-435f-a75b-bf6b103ec8e1","Type":"ContainerStarted","Data":"a29b9d32159a1c943179539fec31cb3b74701663e30a7f27781be632f9de254f"}
May 20 13:50:59.361888 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:59.361861    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:50:59.362345 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:59.361861    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:50:59.362345 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:59.361968    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clw2n" podUID="31842022-6a01-49b0-8b26-fb0d882cdf6f"
May 20 13:50:59.362345 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:50:59.362026    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhcx7" podUID="6969002d-0b97-4cb6-ae5b-03d5ef9dddf8"
May 20 13:50:59.467910 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:59.467774    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 13:50:59.468318 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:59.468126    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5" event={"ID":"56992f6a-130b-435f-a75b-bf6b103ec8e1","Type":"ContainerStarted","Data":"2fe1f2fac88146e89270821d8cf71366744921b3e45197a1e844a21395865f45"}
May 20 13:50:59.468750 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:59.468439    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:50:59.468750 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:59.468673    2570 scope.go:117] "RemoveContainer" containerID="f9375f77ce26401a984567c70bb5b91ea132dd9b92f7b8387eb1dcec4a873ef6"
May 20 13:50:59.486050 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:50:59.485928    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:51:00.473087 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:00.472911    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 13:51:00.473765 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:00.473412    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5" event={"ID":"56992f6a-130b-435f-a75b-bf6b103ec8e1","Type":"ContainerStarted","Data":"e961dfeb615ba5c99ade4d89ecd8294c828088a38d4e4a9f449949ceb028963b"}
May 20 13:51:00.473765 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:00.473633    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:51:00.473765 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:00.473658    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:51:00.475056 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:00.475030    2570 generic.go:358] "Generic (PLEG): container finished" podID="ead02d78-f55b-4612-bb8d-057d70f6ecc4" containerID="53dd6327f816a01f2e9ebb71b91267048cfd1f943842c971ea0237a33f1a74c0" exitCode=0
May 20 13:51:00.475185 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:00.475059    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zgj4t" event={"ID":"ead02d78-f55b-4612-bb8d-057d70f6ecc4","Type":"ContainerDied","Data":"53dd6327f816a01f2e9ebb71b91267048cfd1f943842c971ea0237a33f1a74c0"}
May 20 13:51:00.489966 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:00.489946    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:51:00.558016 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:00.557962    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5" podStartSLOduration=9.22713938 podStartE2EDuration="27.55794182s" podCreationTimestamp="2026-05-20 13:50:33 +0000 UTC" firstStartedPulling="2026-05-20 13:50:36.086026391 +0000 UTC m=+3.323643015" lastFinishedPulling="2026-05-20 13:50:54.416828822 +0000 UTC m=+21.654445455" observedRunningTime="2026-05-20 13:51:00.5218134 +0000 UTC m=+27.759430045" watchObservedRunningTime="2026-05-20 13:51:00.55794182 +0000 UTC m=+27.795558465"
May 20 13:51:01.359344 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:01.359310    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:51:01.359637 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:51:01.359422    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhcx7" podUID="6969002d-0b97-4cb6-ae5b-03d5ef9dddf8"
May 20 13:51:01.359637 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:01.359481    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:51:01.359637 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:51:01.359588    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clw2n" podUID="31842022-6a01-49b0-8b26-fb0d882cdf6f"
May 20 13:51:02.481325 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:02.481288    2570 generic.go:358] "Generic (PLEG): container finished" podID="ead02d78-f55b-4612-bb8d-057d70f6ecc4" containerID="1dce5af2fb7253544b1174712a91a262cc95b9e3ed06b7ea6f74095172314a3e" exitCode=0
May 20 13:51:02.481810 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:02.481379    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zgj4t" event={"ID":"ead02d78-f55b-4612-bb8d-057d70f6ecc4","Type":"ContainerDied","Data":"1dce5af2fb7253544b1174712a91a262cc95b9e3ed06b7ea6f74095172314a3e"}
May 20 13:51:03.362048 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:03.361762    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:51:03.362251 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:03.362114    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:51:03.362251 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:51:03.362089    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhcx7" podUID="6969002d-0b97-4cb6-ae5b-03d5ef9dddf8"
May 20 13:51:03.363354 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:51:03.363328    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clw2n" podUID="31842022-6a01-49b0-8b26-fb0d882cdf6f"
May 20 13:51:04.487426 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:04.487387    2570 generic.go:358] "Generic (PLEG): container finished" podID="ead02d78-f55b-4612-bb8d-057d70f6ecc4" containerID="43c59d0ef2f915c1342b11f800c22bbc8aed8cc70b3a717f40529b3acc923dad" exitCode=0
May 20 13:51:04.487941 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:04.487444    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zgj4t" event={"ID":"ead02d78-f55b-4612-bb8d-057d70f6ecc4","Type":"ContainerDied","Data":"43c59d0ef2f915c1342b11f800c22bbc8aed8cc70b3a717f40529b3acc923dad"}
May 20 13:51:05.359443 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:05.359407    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:51:05.359671 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:51:05.359572    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clw2n" podUID="31842022-6a01-49b0-8b26-fb0d882cdf6f"
May 20 13:51:05.359671 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:05.359625    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:51:05.359796 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:51:05.359740    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhcx7" podUID="6969002d-0b97-4cb6-ae5b-03d5ef9dddf8"
May 20 13:51:07.083846 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:07.083813    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wpd62\" (UniqueName: \"kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62\") pod \"network-check-target-clw2n\" (UID: \"31842022-6a01-49b0-8b26-fb0d882cdf6f\") " pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:51:07.084273 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:07.083886    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs\") pod \"network-metrics-daemon-rhcx7\" (UID: \"6969002d-0b97-4cb6-ae5b-03d5ef9dddf8\") " pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:51:07.084273 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:51:07.083982    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:51:07.084273 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:51:07.083997    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:51:07.084273 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:51:07.084006    2570 projected.go:194] Error preparing data for projected volume kube-api-access-wpd62 for pod openshift-network-diagnostics/network-check-target-clw2n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:51:07.084273 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:51:07.084029    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:51:07.084273 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:51:07.084064    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62 podName:31842022-6a01-49b0-8b26-fb0d882cdf6f nodeName:}" failed. No retries permitted until 2026-05-20 13:51:39.084046758 +0000 UTC m=+66.321663384 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wpd62" (UniqueName: "kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62") pod "network-check-target-clw2n" (UID: "31842022-6a01-49b0-8b26-fb0d882cdf6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:51:07.084273 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:51:07.084101    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs podName:6969002d-0b97-4cb6-ae5b-03d5ef9dddf8 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:39.084080545 +0000 UTC m=+66.321697175 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs") pod "network-metrics-daemon-rhcx7" (UID: "6969002d-0b97-4cb6-ae5b-03d5ef9dddf8") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:51:07.359180 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:07.359107    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:51:07.359430 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:07.359107    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:51:07.359430 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:51:07.359245    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhcx7" podUID="6969002d-0b97-4cb6-ae5b-03d5ef9dddf8"
May 20 13:51:07.359430 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:51:07.359280    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clw2n" podUID="31842022-6a01-49b0-8b26-fb0d882cdf6f"
May 20 13:51:07.704141 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:07.704061    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rhcx7"]
May 20 13:51:07.704376 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:07.704183    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:51:07.704376 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:51:07.704297    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhcx7" podUID="6969002d-0b97-4cb6-ae5b-03d5ef9dddf8"
May 20 13:51:07.708965 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:07.708932    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-clw2n"]
May 20 13:51:07.709083 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:07.709041    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:51:07.709145 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:51:07.709123    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clw2n" podUID="31842022-6a01-49b0-8b26-fb0d882cdf6f"
May 20 13:51:09.358987 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:09.358955    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:51:09.359727 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:09.358991    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:51:09.359727 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:51:09.359092    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhcx7" podUID="6969002d-0b97-4cb6-ae5b-03d5ef9dddf8"
May 20 13:51:09.359727 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:51:09.359242    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clw2n" podUID="31842022-6a01-49b0-8b26-fb0d882cdf6f"
May 20 13:51:11.029067 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.029043    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeReady"
May 20 13:51:11.029535 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.029155    2570 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 20 13:51:11.142079 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.142049    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-np4p2"]
May 20 13:51:11.156471 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.156441    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-np4p2"
May 20 13:51:11.165258 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.165098    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6l4zd\""
May 20 13:51:11.165358 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.165339    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 20 13:51:11.165977 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.165958    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 20 13:51:11.166594 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.166577    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-df47p"]
May 20 13:51:11.172499 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.172478    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 20 13:51:11.184623 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.184584    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-np4p2"]
May 20 13:51:11.184739 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.184666    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-df47p"
May 20 13:51:11.188958 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.188942    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8jz5j\""
May 20 13:51:11.194154 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.194137    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 20 13:51:11.195753 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.195731    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-df47p"]
May 20 13:51:11.200745 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.200726    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 20 13:51:11.317045 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.317009    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92llh\" (UniqueName: \"kubernetes.io/projected/030d660b-7832-4b43-bf8f-d280dc34a607-kube-api-access-92llh\") pod \"ingress-canary-np4p2\" (UID: \"030d660b-7832-4b43-bf8f-d280dc34a607\") " pod="openshift-ingress-canary/ingress-canary-np4p2"
May 20 13:51:11.317045 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.317047    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bb7fe7c-d50a-4d19-8be0-ded9425983e3-metrics-tls\") pod \"dns-default-df47p\" (UID: \"0bb7fe7c-d50a-4d19-8be0-ded9425983e3\") " pod="openshift-dns/dns-default-df47p"
May 20 13:51:11.317273 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.317069    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bb7fe7c-d50a-4d19-8be0-ded9425983e3-tmp-dir\") pod \"dns-default-df47p\" (UID: \"0bb7fe7c-d50a-4d19-8be0-ded9425983e3\") " pod="openshift-dns/dns-default-df47p"
May 20 13:51:11.317273 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.317087    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/030d660b-7832-4b43-bf8f-d280dc34a607-cert\") pod \"ingress-canary-np4p2\" (UID: \"030d660b-7832-4b43-bf8f-d280dc34a607\") " pod="openshift-ingress-canary/ingress-canary-np4p2"
May 20 13:51:11.317273 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.317166    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0bb7fe7c-d50a-4d19-8be0-ded9425983e3-config-volume\") pod \"dns-default-df47p\" (UID: \"0bb7fe7c-d50a-4d19-8be0-ded9425983e3\") " pod="openshift-dns/dns-default-df47p"
May 20 13:51:11.317273 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.317206    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccn9l\" (UniqueName: \"kubernetes.io/projected/0bb7fe7c-d50a-4d19-8be0-ded9425983e3-kube-api-access-ccn9l\") pod \"dns-default-df47p\" (UID: \"0bb7fe7c-d50a-4d19-8be0-ded9425983e3\") " pod="openshift-dns/dns-default-df47p"
May 20 13:51:11.359302 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.359268    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:51:11.359302 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.359294    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:51:11.365535 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.365515    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 20 13:51:11.365535 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.365529    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-68ths\""
May 20 13:51:11.365799 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.365783    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qph4s\""
May 20 13:51:11.365852 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.365829    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 20 13:51:11.367066 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.367051    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 20 13:51:11.417666 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.417607    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0bb7fe7c-d50a-4d19-8be0-ded9425983e3-config-volume\") pod \"dns-default-df47p\" (UID: \"0bb7fe7c-d50a-4d19-8be0-ded9425983e3\") " pod="openshift-dns/dns-default-df47p"
May 20 13:51:11.417849 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.417693    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ccn9l\" (UniqueName: \"kubernetes.io/projected/0bb7fe7c-d50a-4d19-8be0-ded9425983e3-kube-api-access-ccn9l\") pod \"dns-default-df47p\" (UID: \"0bb7fe7c-d50a-4d19-8be0-ded9425983e3\") " pod="openshift-dns/dns-default-df47p"
May 20 13:51:11.417849 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.417745    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-92llh\" (UniqueName: \"kubernetes.io/projected/030d660b-7832-4b43-bf8f-d280dc34a607-kube-api-access-92llh\") pod \"ingress-canary-np4p2\" (UID: \"030d660b-7832-4b43-bf8f-d280dc34a607\") " pod="openshift-ingress-canary/ingress-canary-np4p2"
May 20 13:51:11.417849 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.417769    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bb7fe7c-d50a-4d19-8be0-ded9425983e3-metrics-tls\") pod \"dns-default-df47p\" (UID: \"0bb7fe7c-d50a-4d19-8be0-ded9425983e3\") " pod="openshift-dns/dns-default-df47p"
May 20 13:51:11.417849 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.417799    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bb7fe7c-d50a-4d19-8be0-ded9425983e3-tmp-dir\") pod \"dns-default-df47p\" (UID: \"0bb7fe7c-d50a-4d19-8be0-ded9425983e3\") " pod="openshift-dns/dns-default-df47p"
May 20 13:51:11.417849 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.417824    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/030d660b-7832-4b43-bf8f-d280dc34a607-cert\") pod \"ingress-canary-np4p2\" (UID: \"030d660b-7832-4b43-bf8f-d280dc34a607\") " pod="openshift-ingress-canary/ingress-canary-np4p2"
May 20 13:51:11.418184 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.418164    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bb7fe7c-d50a-4d19-8be0-ded9425983e3-tmp-dir\") pod \"dns-default-df47p\" (UID: \"0bb7fe7c-d50a-4d19-8be0-ded9425983e3\") " pod="openshift-dns/dns-default-df47p"
May 20 13:51:11.418275 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.418260    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0bb7fe7c-d50a-4d19-8be0-ded9425983e3-config-volume\") pod \"dns-default-df47p\" (UID: \"0bb7fe7c-d50a-4d19-8be0-ded9425983e3\") " pod="openshift-dns/dns-default-df47p"
May 20 13:51:11.421926 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.421904    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bb7fe7c-d50a-4d19-8be0-ded9425983e3-metrics-tls\") pod \"dns-default-df47p\" (UID: \"0bb7fe7c-d50a-4d19-8be0-ded9425983e3\") " pod="openshift-dns/dns-default-df47p"
May 20 13:51:11.422120 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.422103    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/030d660b-7832-4b43-bf8f-d280dc34a607-cert\") pod \"ingress-canary-np4p2\" (UID: \"030d660b-7832-4b43-bf8f-d280dc34a607\") " pod="openshift-ingress-canary/ingress-canary-np4p2"
May 20 13:51:11.445866 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.445836    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccn9l\" (UniqueName: \"kubernetes.io/projected/0bb7fe7c-d50a-4d19-8be0-ded9425983e3-kube-api-access-ccn9l\") pod \"dns-default-df47p\" (UID: \"0bb7fe7c-d50a-4d19-8be0-ded9425983e3\") " pod="openshift-dns/dns-default-df47p"
May 20 13:51:11.456336 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.456302    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92llh\" (UniqueName: \"kubernetes.io/projected/030d660b-7832-4b43-bf8f-d280dc34a607-kube-api-access-92llh\") pod \"ingress-canary-np4p2\" (UID: \"030d660b-7832-4b43-bf8f-d280dc34a607\") " pod="openshift-ingress-canary/ingress-canary-np4p2"
May 20 13:51:11.465952 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.465926    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-np4p2"
May 20 13:51:11.492793 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.492743    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-df47p"
May 20 13:51:11.504317 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.504057    2570 generic.go:358] "Generic (PLEG): container finished" podID="ead02d78-f55b-4612-bb8d-057d70f6ecc4" containerID="0acb9607abfb9c4642ac3c3b69a2d0b909348bb699f8941ba5f62fe2325f906f" exitCode=0
May 20 13:51:11.504317 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.504125    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zgj4t" event={"ID":"ead02d78-f55b-4612-bb8d-057d70f6ecc4","Type":"ContainerDied","Data":"0acb9607abfb9c4642ac3c3b69a2d0b909348bb699f8941ba5f62fe2325f906f"}
May 20 13:51:11.696968 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.696940    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-np4p2"]
May 20 13:51:11.700020 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:51:11.699990    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030d660b_7832_4b43_bf8f_d280dc34a607.slice/crio-f518a62b4d5dfc6240e94c0ec828bcffa4a9c4a4c041449ad0ae476fcd9efebf WatchSource:0}: Error finding container f518a62b4d5dfc6240e94c0ec828bcffa4a9c4a4c041449ad0ae476fcd9efebf: Status 404 returned error can't find the container with id f518a62b4d5dfc6240e94c0ec828bcffa4a9c4a4c041449ad0ae476fcd9efebf
May 20 13:51:11.705028 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.704997    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-df47p"]
May 20 13:51:11.738444 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:11.738423    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cs8gn_b985f05d-7aa5-42b7-a5d7-3b18a2769356/dns-node-resolver/0.log"
May 20 13:51:12.510881 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:12.510084    2570 generic.go:358] "Generic (PLEG): container finished" podID="ead02d78-f55b-4612-bb8d-057d70f6ecc4" containerID="577607beb3286e4e685d5c829b73f9b6f489729aa2e9c253d56133cd0a439e4f" exitCode=0
May 20 13:51:12.510881 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:12.510165    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zgj4t" event={"ID":"ead02d78-f55b-4612-bb8d-057d70f6ecc4","Type":"ContainerDied","Data":"577607beb3286e4e685d5c829b73f9b6f489729aa2e9c253d56133cd0a439e4f"}
May 20 13:51:12.512024 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:12.511513    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-np4p2" event={"ID":"030d660b-7832-4b43-bf8f-d280dc34a607","Type":"ContainerStarted","Data":"f518a62b4d5dfc6240e94c0ec828bcffa4a9c4a4c041449ad0ae476fcd9efebf"}
May 20 13:51:12.513814 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:12.513792    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-df47p" event={"ID":"0bb7fe7c-d50a-4d19-8be0-ded9425983e3","Type":"ContainerStarted","Data":"b2729f86d13002674f3d8cee48e8b884664020beb4ce7e7fa76efa7c070b6044"}
May 20 13:51:12.837192 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:12.836980    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dk92v_efd2c3ea-0867-4025-b5c6-03b9a0d744c5/node-ca/0.log"
May 20 13:51:13.520681 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:13.520598    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zgj4t" event={"ID":"ead02d78-f55b-4612-bb8d-057d70f6ecc4","Type":"ContainerStarted","Data":"30f979b3bf1088bc77856c6a1fcc35b1f83c6270c805772369583acfdfbe20c8"}
May 20 13:51:13.566009 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:13.565954    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zgj4t" podStartSLOduration=6.142441516 podStartE2EDuration="40.565936227s" podCreationTimestamp="2026-05-20 13:50:33 +0000 UTC" firstStartedPulling="2026-05-20 13:50:36.076370441 +0000 UTC m=+3.313987081" lastFinishedPulling="2026-05-20 13:51:10.499865167 +0000 UTC m=+37.737481792" observedRunningTime="2026-05-20 13:51:13.565907442 +0000 UTC m=+40.803524110" watchObservedRunningTime="2026-05-20 13:51:13.565936227 +0000 UTC m=+40.803552873"
May 20 13:51:14.308455 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.308418    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-2qf4d"]
May 20 13:51:14.310290 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.310268    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-2qf4d"
May 20 13:51:14.318053 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.318029    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 20 13:51:14.318234 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.318064    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 20 13:51:14.318961 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.318947    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-962j6\""
May 20 13:51:14.319072 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.319058    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 20 13:51:14.319132 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.319057    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 20 13:51:14.319185 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.319136    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 20 13:51:14.327101 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.327075    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-2qf4d"]
May 20 13:51:14.441530 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.441498    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2935ba18-80fa-4e6f-b4c1-6fe81a9b7981-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-2qf4d\" (UID: \"2935ba18-80fa-4e6f-b4c1-6fe81a9b7981\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-2qf4d"
May 20 13:51:14.441728 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.441549    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2935ba18-80fa-4e6f-b4c1-6fe81a9b7981-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-2qf4d\" (UID: \"2935ba18-80fa-4e6f-b4c1-6fe81a9b7981\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-2qf4d"
May 20 13:51:14.441728 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.441663    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2935ba18-80fa-4e6f-b4c1-6fe81a9b7981-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-2qf4d\" (UID: \"2935ba18-80fa-4e6f-b4c1-6fe81a9b7981\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-2qf4d"
May 20 13:51:14.441728 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.441694    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkt98\" (UniqueName: \"kubernetes.io/projected/2935ba18-80fa-4e6f-b4c1-6fe81a9b7981-kube-api-access-jkt98\") pod \"prometheus-operator-7779bbdd78-2qf4d\" (UID: \"2935ba18-80fa-4e6f-b4c1-6fe81a9b7981\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-2qf4d"
May 20 13:51:14.524686 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.524651    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-df47p" event={"ID":"0bb7fe7c-d50a-4d19-8be0-ded9425983e3","Type":"ContainerStarted","Data":"d9efa9809d2793259603bfb97a1fa47ba3856b8565c9f4fdbd8f459947b044c0"}
May 20 13:51:14.524686 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.524685    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-df47p" event={"ID":"0bb7fe7c-d50a-4d19-8be0-ded9425983e3","Type":"ContainerStarted","Data":"855148bf8d3ba88a32cf91b22c07fbe55562a946299e510e9e60e5abf18fde6e"}
May 20 13:51:14.525167 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.524768    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-df47p"
May 20 13:51:14.525911 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.525886    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-np4p2" event={"ID":"030d660b-7832-4b43-bf8f-d280dc34a607","Type":"ContainerStarted","Data":"ca0e7acb883a2ea28d98e6639c4e729e0a180dcdd9439dc64cfd7df769c1fdf8"}
May 20 13:51:14.542711 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.542689    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2935ba18-80fa-4e6f-b4c1-6fe81a9b7981-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-2qf4d\" (UID: \"2935ba18-80fa-4e6f-b4c1-6fe81a9b7981\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-2qf4d"
May 20 13:51:14.542781 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.542716    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jkt98\" (UniqueName: \"kubernetes.io/projected/2935ba18-80fa-4e6f-b4c1-6fe81a9b7981-kube-api-access-jkt98\") pod \"prometheus-operator-7779bbdd78-2qf4d\" (UID: \"2935ba18-80fa-4e6f-b4c1-6fe81a9b7981\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-2qf4d"
May 20 13:51:14.542781 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.542765    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2935ba18-80fa-4e6f-b4c1-6fe81a9b7981-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-2qf4d\" (UID: \"2935ba18-80fa-4e6f-b4c1-6fe81a9b7981\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-2qf4d"
May 20 13:51:14.542850 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.542809    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2935ba18-80fa-4e6f-b4c1-6fe81a9b7981-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-2qf4d\" (UID: \"2935ba18-80fa-4e6f-b4c1-6fe81a9b7981\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-2qf4d"
May 20 13:51:14.543418 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.543395    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2935ba18-80fa-4e6f-b4c1-6fe81a9b7981-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-2qf4d\" (UID: \"2935ba18-80fa-4e6f-b4c1-6fe81a9b7981\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-2qf4d"
May 20 13:51:14.546557 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.546535    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2935ba18-80fa-4e6f-b4c1-6fe81a9b7981-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-2qf4d\" (UID: \"2935ba18-80fa-4e6f-b4c1-6fe81a9b7981\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-2qf4d"
May 20 13:51:14.546676 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.546558    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2935ba18-80fa-4e6f-b4c1-6fe81a9b7981-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-2qf4d\" (UID: \"2935ba18-80fa-4e6f-b4c1-6fe81a9b7981\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-2qf4d"
May 20 13:51:14.555196 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.555163    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkt98\" (UniqueName: \"kubernetes.io/projected/2935ba18-80fa-4e6f-b4c1-6fe81a9b7981-kube-api-access-jkt98\") pod \"prometheus-operator-7779bbdd78-2qf4d\" (UID: \"2935ba18-80fa-4e6f-b4c1-6fe81a9b7981\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-2qf4d"
May 20 13:51:14.561107 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.561037    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-df47p" podStartSLOduration=1.7978039689999998 podStartE2EDuration="3.561026606s" podCreationTimestamp="2026-05-20 13:51:11 +0000 UTC" firstStartedPulling="2026-05-20 13:51:11.711012065 +0000 UTC m=+38.948628689" lastFinishedPulling="2026-05-20 13:51:13.474234702 +0000 UTC m=+40.711851326" observedRunningTime="2026-05-20 13:51:14.558655714 +0000 UTC m=+41.796272359" watchObservedRunningTime="2026-05-20 13:51:14.561026606 +0000 UTC m=+41.798643251"
May 20 13:51:14.618859 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.618830    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-2qf4d"
May 20 13:51:14.759164 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.759114    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-np4p2" podStartSLOduration=1.984784739 podStartE2EDuration="3.759098507s" podCreationTimestamp="2026-05-20 13:51:11 +0000 UTC" firstStartedPulling="2026-05-20 13:51:11.70188317 +0000 UTC m=+38.939499795" lastFinishedPulling="2026-05-20 13:51:13.476196936 +0000 UTC m=+40.713813563" observedRunningTime="2026-05-20 13:51:14.589739341 +0000 UTC m=+41.827355986" watchObservedRunningTime="2026-05-20 13:51:14.759098507 +0000 UTC m=+41.996715163"
May 20 13:51:14.759566 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:14.759530    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-2qf4d"]
May 20 13:51:14.763089 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:51:14.763061    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2935ba18_80fa_4e6f_b4c1_6fe81a9b7981.slice/crio-f991efd9083d785353da0b3ce8fe059645dffbc4063ce04716d227111928887a WatchSource:0}: Error finding container f991efd9083d785353da0b3ce8fe059645dffbc4063ce04716d227111928887a: Status 404 returned error can't find the container with id f991efd9083d785353da0b3ce8fe059645dffbc4063ce04716d227111928887a
May 20 13:51:15.529573 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:15.529529    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-2qf4d" event={"ID":"2935ba18-80fa-4e6f-b4c1-6fe81a9b7981","Type":"ContainerStarted","Data":"f991efd9083d785353da0b3ce8fe059645dffbc4063ce04716d227111928887a"}
May 20 13:51:16.534487 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:16.534450    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-2qf4d" event={"ID":"2935ba18-80fa-4e6f-b4c1-6fe81a9b7981","Type":"ContainerStarted","Data":"b8f678411e5610dca4e319a066aa100f49ad4e2a3f2574ac30a32970c80615ae"}
May 20 13:51:16.534487 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:16.534488    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-2qf4d" event={"ID":"2935ba18-80fa-4e6f-b4c1-6fe81a9b7981","Type":"ContainerStarted","Data":"fcf559127f25bf1939ec1d0072930ac26d5e2bc0fb65a911c6a980c5d987e410"}
May 20 13:51:16.562077 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:16.562030    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-2qf4d" podStartSLOduration=1.293846382 podStartE2EDuration="2.562017247s" podCreationTimestamp="2026-05-20 13:51:14 +0000 UTC" firstStartedPulling="2026-05-20 13:51:14.765461094 +0000 UTC m=+42.003077719" lastFinishedPulling="2026-05-20 13:51:16.033631955 +0000 UTC m=+43.271248584" observedRunningTime="2026-05-20 13:51:16.561826792 +0000 UTC m=+43.799443437" watchObservedRunningTime="2026-05-20 13:51:16.562017247 +0000 UTC m=+43.799633892"
May 20 13:51:17.046359 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.046327    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-579cb6c44-k9mx7"]
May 20 13:51:17.048186 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.048169    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:17.052310 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.052291    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 20 13:51:17.053348 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.053303    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 20 13:51:17.053348 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.053328    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 20 13:51:17.053495 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.053375    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 20 13:51:17.053495 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.053397    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-k5649\""
May 20 13:51:17.053495 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.053400    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 20 13:51:17.053495 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.053434    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 20 13:51:17.053717 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.053703    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 20 13:51:17.062310 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.062288    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81b01ffe-be78-4c08-8ac6-1a722d9ad441-service-ca\") pod \"console-579cb6c44-k9mx7\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") " pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:17.062405 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.062319    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81b01ffe-be78-4c08-8ac6-1a722d9ad441-oauth-serving-cert\") pod \"console-579cb6c44-k9mx7\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") " pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:17.062405 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.062354    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81b01ffe-be78-4c08-8ac6-1a722d9ad441-console-config\") pod \"console-579cb6c44-k9mx7\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") " pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:17.062405 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.062375    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81b01ffe-be78-4c08-8ac6-1a722d9ad441-console-serving-cert\") pod \"console-579cb6c44-k9mx7\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") " pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:17.062405 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.062391    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr2wz\" (UniqueName: \"kubernetes.io/projected/81b01ffe-be78-4c08-8ac6-1a722d9ad441-kube-api-access-gr2wz\") pod \"console-579cb6c44-k9mx7\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") " pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:17.062568 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.062412    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81b01ffe-be78-4c08-8ac6-1a722d9ad441-console-oauth-config\") pod \"console-579cb6c44-k9mx7\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") " pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:17.069150 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.069125    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-579cb6c44-k9mx7"]
May 20 13:51:17.163222 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.163189    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gr2wz\" (UniqueName: \"kubernetes.io/projected/81b01ffe-be78-4c08-8ac6-1a722d9ad441-kube-api-access-gr2wz\") pod \"console-579cb6c44-k9mx7\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") " pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:17.163222 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.163230    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81b01ffe-be78-4c08-8ac6-1a722d9ad441-console-oauth-config\") pod \"console-579cb6c44-k9mx7\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") " pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:17.163471 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.163272    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81b01ffe-be78-4c08-8ac6-1a722d9ad441-service-ca\") pod \"console-579cb6c44-k9mx7\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") " pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:17.163471 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.163286    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81b01ffe-be78-4c08-8ac6-1a722d9ad441-oauth-serving-cert\") pod \"console-579cb6c44-k9mx7\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") " pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:17.163471 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.163328    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81b01ffe-be78-4c08-8ac6-1a722d9ad441-console-config\") pod \"console-579cb6c44-k9mx7\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") " pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:17.163471 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.163348    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81b01ffe-be78-4c08-8ac6-1a722d9ad441-console-serving-cert\") pod \"console-579cb6c44-k9mx7\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") " pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:17.164227 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.164194    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81b01ffe-be78-4c08-8ac6-1a722d9ad441-oauth-serving-cert\") pod \"console-579cb6c44-k9mx7\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") " pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:17.164227 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.164197    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81b01ffe-be78-4c08-8ac6-1a722d9ad441-console-config\") pod \"console-579cb6c44-k9mx7\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") " pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:17.164372 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.164252    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81b01ffe-be78-4c08-8ac6-1a722d9ad441-service-ca\") pod \"console-579cb6c44-k9mx7\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") " pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:17.165890 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.165869    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81b01ffe-be78-4c08-8ac6-1a722d9ad441-console-oauth-config\") pod \"console-579cb6c44-k9mx7\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") " pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:17.165890 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.165886    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81b01ffe-be78-4c08-8ac6-1a722d9ad441-console-serving-cert\") pod \"console-579cb6c44-k9mx7\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") " pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:17.208376 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.208349    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr2wz\" (UniqueName: \"kubernetes.io/projected/81b01ffe-be78-4c08-8ac6-1a722d9ad441-kube-api-access-gr2wz\") pod \"console-579cb6c44-k9mx7\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") " pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:17.214983 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.214955    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-n9979"]
May 20 13:51:17.217264 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.217249    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n9979"
May 20 13:51:17.221804 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.221782    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 20 13:51:17.221966 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.221948    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 20 13:51:17.222218 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.222201    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dmm4n\""
May 20 13:51:17.222445 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.222432    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 20 13:51:17.222969 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.222956    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 20 13:51:17.232722 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.232700    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n9979"]
May 20 13:51:17.264237 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.264145    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e95098e3-08f2-4619-8504-9cdc39ae3b80-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n9979\" (UID: \"e95098e3-08f2-4619-8504-9cdc39ae3b80\") " pod="openshift-insights/insights-runtime-extractor-n9979"
May 20 13:51:17.264237 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.264183    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e95098e3-08f2-4619-8504-9cdc39ae3b80-crio-socket\") pod \"insights-runtime-extractor-n9979\" (UID: \"e95098e3-08f2-4619-8504-9cdc39ae3b80\") " pod="openshift-insights/insights-runtime-extractor-n9979"
May 20 13:51:17.264237 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.264213    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e95098e3-08f2-4619-8504-9cdc39ae3b80-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n9979\" (UID: \"e95098e3-08f2-4619-8504-9cdc39ae3b80\") " pod="openshift-insights/insights-runtime-extractor-n9979"
May 20 13:51:17.264477 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.264290    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r55kp\" (UniqueName: \"kubernetes.io/projected/e95098e3-08f2-4619-8504-9cdc39ae3b80-kube-api-access-r55kp\") pod \"insights-runtime-extractor-n9979\" (UID: \"e95098e3-08f2-4619-8504-9cdc39ae3b80\") " pod="openshift-insights/insights-runtime-extractor-n9979"
May 20 13:51:17.264477 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.264328    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e95098e3-08f2-4619-8504-9cdc39ae3b80-data-volume\") pod \"insights-runtime-extractor-n9979\" (UID: \"e95098e3-08f2-4619-8504-9cdc39ae3b80\") " pod="openshift-insights/insights-runtime-extractor-n9979"
May 20 13:51:17.356288 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.356244    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:17.364718 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.364679    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r55kp\" (UniqueName: \"kubernetes.io/projected/e95098e3-08f2-4619-8504-9cdc39ae3b80-kube-api-access-r55kp\") pod \"insights-runtime-extractor-n9979\" (UID: \"e95098e3-08f2-4619-8504-9cdc39ae3b80\") " pod="openshift-insights/insights-runtime-extractor-n9979"
May 20 13:51:17.364801 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.364734    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e95098e3-08f2-4619-8504-9cdc39ae3b80-data-volume\") pod \"insights-runtime-extractor-n9979\" (UID: \"e95098e3-08f2-4619-8504-9cdc39ae3b80\") " pod="openshift-insights/insights-runtime-extractor-n9979"
May 20 13:51:17.364850 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.364814    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e95098e3-08f2-4619-8504-9cdc39ae3b80-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n9979\" (UID: \"e95098e3-08f2-4619-8504-9cdc39ae3b80\") " pod="openshift-insights/insights-runtime-extractor-n9979"
May 20 13:51:17.364895 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.364847    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e95098e3-08f2-4619-8504-9cdc39ae3b80-crio-socket\") pod \"insights-runtime-extractor-n9979\" (UID: \"e95098e3-08f2-4619-8504-9cdc39ae3b80\") " pod="openshift-insights/insights-runtime-extractor-n9979"
May 20 13:51:17.364954 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.364902    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e95098e3-08f2-4619-8504-9cdc39ae3b80-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n9979\" (UID: \"e95098e3-08f2-4619-8504-9cdc39ae3b80\") " pod="openshift-insights/insights-runtime-extractor-n9979"
May 20 13:51:17.365240 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.365192    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e95098e3-08f2-4619-8504-9cdc39ae3b80-data-volume\") pod \"insights-runtime-extractor-n9979\" (UID: \"e95098e3-08f2-4619-8504-9cdc39ae3b80\") " pod="openshift-insights/insights-runtime-extractor-n9979"
May 20 13:51:17.365340 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.365298    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e95098e3-08f2-4619-8504-9cdc39ae3b80-crio-socket\") pod \"insights-runtime-extractor-n9979\" (UID: \"e95098e3-08f2-4619-8504-9cdc39ae3b80\") " pod="openshift-insights/insights-runtime-extractor-n9979"
May 20 13:51:17.365780 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.365706    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e95098e3-08f2-4619-8504-9cdc39ae3b80-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n9979\" (UID: \"e95098e3-08f2-4619-8504-9cdc39ae3b80\") " pod="openshift-insights/insights-runtime-extractor-n9979"
May 20 13:51:17.367861 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.367836    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e95098e3-08f2-4619-8504-9cdc39ae3b80-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n9979\" (UID: \"e95098e3-08f2-4619-8504-9cdc39ae3b80\") " pod="openshift-insights/insights-runtime-extractor-n9979"
May 20 13:51:17.383035 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.383008    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r55kp\" (UniqueName: \"kubernetes.io/projected/e95098e3-08f2-4619-8504-9cdc39ae3b80-kube-api-access-r55kp\") pod \"insights-runtime-extractor-n9979\" (UID: \"e95098e3-08f2-4619-8504-9cdc39ae3b80\") " pod="openshift-insights/insights-runtime-extractor-n9979"
May 20 13:51:17.524334 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.524297    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-579cb6c44-k9mx7"]
May 20 13:51:17.525456 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.525446    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n9979"
May 20 13:51:17.532376 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:51:17.532348    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b01ffe_be78_4c08_8ac6_1a722d9ad441.slice/crio-42677761c74ff7a960fe3ccbf7daf9c8b8b6a93eb813b546d1a1a436ecdc9fd9 WatchSource:0}: Error finding container 42677761c74ff7a960fe3ccbf7daf9c8b8b6a93eb813b546d1a1a436ecdc9fd9: Status 404 returned error can't find the container with id 42677761c74ff7a960fe3ccbf7daf9c8b8b6a93eb813b546d1a1a436ecdc9fd9
May 20 13:51:17.537385 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.537353    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579cb6c44-k9mx7" event={"ID":"81b01ffe-be78-4c08-8ac6-1a722d9ad441","Type":"ContainerStarted","Data":"42677761c74ff7a960fe3ccbf7daf9c8b8b6a93eb813b546d1a1a436ecdc9fd9"}
May 20 13:51:17.671829 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:17.671784    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n9979"]
May 20 13:51:17.675038 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:51:17.674998    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95098e3_08f2_4619_8504_9cdc39ae3b80.slice/crio-a9e389bbd8ec5a6afc9f211ad67a1c2bece4fd5d9689a761252ba77555c04590 WatchSource:0}: Error finding container a9e389bbd8ec5a6afc9f211ad67a1c2bece4fd5d9689a761252ba77555c04590: Status 404 returned error can't find the container with id a9e389bbd8ec5a6afc9f211ad67a1c2bece4fd5d9689a761252ba77555c04590
May 20 13:51:18.542844 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:18.542809    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n9979" event={"ID":"e95098e3-08f2-4619-8504-9cdc39ae3b80","Type":"ContainerStarted","Data":"1efdf207301065013c23c37464d584d0776ffc797b0d7142ca11ba2bc51703af"}
May 20 13:51:18.543241 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:18.542856    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n9979" event={"ID":"e95098e3-08f2-4619-8504-9cdc39ae3b80","Type":"ContainerStarted","Data":"a9e389bbd8ec5a6afc9f211ad67a1c2bece4fd5d9689a761252ba77555c04590"}
May 20 13:51:19.545300 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.545268    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"]
May 20 13:51:19.548335 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.548299    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n9979" event={"ID":"e95098e3-08f2-4619-8504-9cdc39ae3b80","Type":"ContainerStarted","Data":"8f1887e1a0464b7d5c733ce967702f16ad2ccf9d65dc80e7d64edcd812109c3d"}
May 20 13:51:19.548481 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.548430    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"
May 20 13:51:19.558444 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.558426    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
May 20 13:51:19.558995 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.558975    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
May 20 13:51:19.560378 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.560356    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8dvl7\""
May 20 13:51:19.598326 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.598292    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"]
May 20 13:51:19.602113 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.602079    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
May 20 13:51:19.680083 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.680047    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c1998859-b3d0-4755-bdb3-cb4ccb164e16-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-jxt2d\" (UID: \"c1998859-b3d0-4755-bdb3-cb4ccb164e16\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"
May 20 13:51:19.680083 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.680085    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1998859-b3d0-4755-bdb3-cb4ccb164e16-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-jxt2d\" (UID: \"c1998859-b3d0-4755-bdb3-cb4ccb164e16\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"
May 20 13:51:19.680308 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.680109    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c1998859-b3d0-4755-bdb3-cb4ccb164e16-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-jxt2d\" (UID: \"c1998859-b3d0-4755-bdb3-cb4ccb164e16\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"
May 20 13:51:19.680308 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.680196    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tttsg\" (UniqueName: \"kubernetes.io/projected/c1998859-b3d0-4755-bdb3-cb4ccb164e16-kube-api-access-tttsg\") pod \"kube-state-metrics-54fd6bc5c4-jxt2d\" (UID: \"c1998859-b3d0-4755-bdb3-cb4ccb164e16\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"
May 20 13:51:19.680308 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.680235    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1998859-b3d0-4755-bdb3-cb4ccb164e16-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-jxt2d\" (UID: \"c1998859-b3d0-4755-bdb3-cb4ccb164e16\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"
May 20 13:51:19.680308 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.680256    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1998859-b3d0-4755-bdb3-cb4ccb164e16-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-jxt2d\" (UID: \"c1998859-b3d0-4755-bdb3-cb4ccb164e16\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"
May 20 13:51:19.728261 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.728223    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9cc64"]
May 20 13:51:19.731741 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.731714    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.735650 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.735594    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 20 13:51:19.735770 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.735711    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 20 13:51:19.736003 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.735983    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vchmg\""
May 20 13:51:19.736220 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.736199    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 20 13:51:19.781304 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.781259    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-node-exporter-textfile\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.781304 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.781301    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-node-exporter-tls\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.781541 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.781337    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1998859-b3d0-4755-bdb3-cb4ccb164e16-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-jxt2d\" (UID: \"c1998859-b3d0-4755-bdb3-cb4ccb164e16\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"
May 20 13:51:19.781541 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.781387    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c1998859-b3d0-4755-bdb3-cb4ccb164e16-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-jxt2d\" (UID: \"c1998859-b3d0-4755-bdb3-cb4ccb164e16\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"
May 20 13:51:19.781541 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.781416    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c1998859-b3d0-4755-bdb3-cb4ccb164e16-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-jxt2d\" (UID: \"c1998859-b3d0-4755-bdb3-cb4ccb164e16\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"
May 20 13:51:19.781541 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.781449    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjq2r\" (UniqueName: \"kubernetes.io/projected/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-kube-api-access-sjq2r\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.781541 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.781476    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1998859-b3d0-4755-bdb3-cb4ccb164e16-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-jxt2d\" (UID: \"c1998859-b3d0-4755-bdb3-cb4ccb164e16\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"
May 20 13:51:19.781541 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.781508    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-node-exporter-wtmp\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.781979 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.781545    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-sys\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.781979 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.781580    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tttsg\" (UniqueName: \"kubernetes.io/projected/c1998859-b3d0-4755-bdb3-cb4ccb164e16-kube-api-access-tttsg\") pod \"kube-state-metrics-54fd6bc5c4-jxt2d\" (UID: \"c1998859-b3d0-4755-bdb3-cb4ccb164e16\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"
May 20 13:51:19.781979 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.781637    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-metrics-client-ca\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.781979 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.781690    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-root\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.781979 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.781732    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.781979 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.781759    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1998859-b3d0-4755-bdb3-cb4ccb164e16-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-jxt2d\" (UID: \"c1998859-b3d0-4755-bdb3-cb4ccb164e16\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"
May 20 13:51:19.781979 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.781788    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-node-exporter-accelerators-collector-config\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.781979 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.781881    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c1998859-b3d0-4755-bdb3-cb4ccb164e16-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-jxt2d\" (UID: \"c1998859-b3d0-4755-bdb3-cb4ccb164e16\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"
May 20 13:51:19.782301 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.782128    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c1998859-b3d0-4755-bdb3-cb4ccb164e16-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-jxt2d\" (UID: \"c1998859-b3d0-4755-bdb3-cb4ccb164e16\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"
May 20 13:51:19.782301 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.782160    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1998859-b3d0-4755-bdb3-cb4ccb164e16-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-jxt2d\" (UID: \"c1998859-b3d0-4755-bdb3-cb4ccb164e16\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"
May 20 13:51:19.784477 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.784452    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1998859-b3d0-4755-bdb3-cb4ccb164e16-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-jxt2d\" (UID: \"c1998859-b3d0-4755-bdb3-cb4ccb164e16\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"
May 20 13:51:19.784592 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.784578    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1998859-b3d0-4755-bdb3-cb4ccb164e16-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-jxt2d\" (UID: \"c1998859-b3d0-4755-bdb3-cb4ccb164e16\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"
May 20 13:51:19.805827 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.805754    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tttsg\" (UniqueName: \"kubernetes.io/projected/c1998859-b3d0-4755-bdb3-cb4ccb164e16-kube-api-access-tttsg\") pod \"kube-state-metrics-54fd6bc5c4-jxt2d\" (UID: \"c1998859-b3d0-4755-bdb3-cb4ccb164e16\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"
May 20 13:51:19.858460 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.858428    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"
May 20 13:51:19.882349 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.882313    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-metrics-client-ca\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.882504 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.882363    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-root\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.882504 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.882410    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.882504 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.882435    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-node-exporter-accelerators-collector-config\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.882504 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.882470    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-node-exporter-textfile\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.882504 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.882492    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-node-exporter-tls\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.882756 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.882508    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-root\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.882756 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.882544    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sjq2r\" (UniqueName: \"kubernetes.io/projected/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-kube-api-access-sjq2r\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.882756 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.882578    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-node-exporter-wtmp\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.882756 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.882639    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-sys\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.882756 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.882709    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-sys\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.882990 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.882969    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-node-exporter-textfile\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.883043 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.883002    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-metrics-client-ca\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.883086 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.883063    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-node-exporter-accelerators-collector-config\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.883392 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.883369    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-node-exporter-wtmp\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.885723 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.885689    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-node-exporter-tls\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.885827 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.885810    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:19.908988 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:19.908925    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjq2r\" (UniqueName: \"kubernetes.io/projected/a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed-kube-api-access-sjq2r\") pod \"node-exporter-9cc64\" (UID: \"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed\") " pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:20.042324 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.042273    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9cc64"
May 20 13:51:20.906878 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.906823    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:51:20.910373 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.910350    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:20.915304 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.915278    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 20 13:51:20.915435 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.915417    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 20 13:51:20.923875 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.923255    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 20 13:51:20.923875 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.923354    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 20 13:51:20.923875 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.923373    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 20 13:51:20.923875 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.923478    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tclzk\""
May 20 13:51:20.923875 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.923493    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 20 13:51:20.923875 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.923677    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 20 13:51:20.923875 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.923680    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 20 13:51:20.932401 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.932379    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 20 13:51:20.964412 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.964375    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:51:20.993026 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.992995    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4cf4bd0-9845-4f94-bf86-154c26adcb33-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:20.993188 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.993044    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4cf4bd0-9845-4f94-bf86-154c26adcb33-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:20.993188 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.993068    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4cf4bd0-9845-4f94-bf86-154c26adcb33-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:20.993188 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.993094    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-web-config\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:20.993188 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.993141    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-config-volume\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:20.993188 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.993178    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:20.993366 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.993197    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:20.993366 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.993217    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c4cf4bd0-9845-4f94-bf86-154c26adcb33-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:20.993366 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.993262    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4cf4bd0-9845-4f94-bf86-154c26adcb33-config-out\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:20.993366 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.993303    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nct2z\" (UniqueName: \"kubernetes.io/projected/c4cf4bd0-9845-4f94-bf86-154c26adcb33-kube-api-access-nct2z\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:20.993366 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.993321    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:20.993366 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.993339    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:20.993366 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:20.993359    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.057386 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.057347    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d"]
May 20 13:51:21.061974 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:51:21.061949    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1998859_b3d0_4755_bdb3_cb4ccb164e16.slice/crio-c041f0f0fbcc16269c9f558d98a08e6e878058365665401b3a3df915fa8ecbe3 WatchSource:0}: Error finding container c041f0f0fbcc16269c9f558d98a08e6e878058365665401b3a3df915fa8ecbe3: Status 404 returned error can't find the container with id c041f0f0fbcc16269c9f558d98a08e6e878058365665401b3a3df915fa8ecbe3
May 20 13:51:21.093956 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.093928    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nct2z\" (UniqueName: \"kubernetes.io/projected/c4cf4bd0-9845-4f94-bf86-154c26adcb33-kube-api-access-nct2z\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.094222 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.093965    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.094222 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.093986    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.094222 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.094009    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.094222 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.094040    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4cf4bd0-9845-4f94-bf86-154c26adcb33-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.094222 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.094072    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4cf4bd0-9845-4f94-bf86-154c26adcb33-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.094222 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.094094    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4cf4bd0-9845-4f94-bf86-154c26adcb33-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.094222 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.094116    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-web-config\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.094222 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.094141    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-config-volume\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.094222 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.094169    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.094222 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.094195    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.094724 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.094238    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c4cf4bd0-9845-4f94-bf86-154c26adcb33-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.094724 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.094275    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4cf4bd0-9845-4f94-bf86-154c26adcb33-config-out\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.095901 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.095498    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c4cf4bd0-9845-4f94-bf86-154c26adcb33-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.095901 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.095759    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4cf4bd0-9845-4f94-bf86-154c26adcb33-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.096322 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.096003    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4cf4bd0-9845-4f94-bf86-154c26adcb33-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.097439 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.097395    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.097439 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.097407    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.097585 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.097483    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.097650 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.097625    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4cf4bd0-9845-4f94-bf86-154c26adcb33-config-out\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.097960 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.097938    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4cf4bd0-9845-4f94-bf86-154c26adcb33-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.098003 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.097965    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.098662 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.098639    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-web-config\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.098777 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.098760    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-config-volume\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.099249 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.099233    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.134817 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.134787    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nct2z\" (UniqueName: \"kubernetes.io/projected/c4cf4bd0-9845-4f94-bf86-154c26adcb33-kube-api-access-nct2z\") pod \"alertmanager-main-0\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.224283 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.224248    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 13:51:21.382119 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.382093    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:51:21.384460 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:51:21.384428    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4cf4bd0_9845_4f94_bf86_154c26adcb33.slice/crio-6f3fc791656ca153ff8b31fc25c0d0bb958b6dd70aa370cb9b098674f9b03f32 WatchSource:0}: Error finding container 6f3fc791656ca153ff8b31fc25c0d0bb958b6dd70aa370cb9b098674f9b03f32: Status 404 returned error can't find the container with id 6f3fc791656ca153ff8b31fc25c0d0bb958b6dd70aa370cb9b098674f9b03f32
May 20 13:51:21.555240 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.555190    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4cf4bd0-9845-4f94-bf86-154c26adcb33","Type":"ContainerStarted","Data":"6f3fc791656ca153ff8b31fc25c0d0bb958b6dd70aa370cb9b098674f9b03f32"}
May 20 13:51:21.558298 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.557454    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n9979" event={"ID":"e95098e3-08f2-4619-8504-9cdc39ae3b80","Type":"ContainerStarted","Data":"b580491642afaae262fec36f2474e24139289d38cd072e8d1cbe1008a26a4737"}
May 20 13:51:21.559988 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.559961    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579cb6c44-k9mx7" event={"ID":"81b01ffe-be78-4c08-8ac6-1a722d9ad441","Type":"ContainerStarted","Data":"b17ba85ebad631589b9e1cfe4fec2d2156b45ca5171661f9dc88a5ec109d2b02"}
May 20 13:51:21.561315 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.561286    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d" event={"ID":"c1998859-b3d0-4755-bdb3-cb4ccb164e16","Type":"ContainerStarted","Data":"c041f0f0fbcc16269c9f558d98a08e6e878058365665401b3a3df915fa8ecbe3"}
May 20 13:51:21.562458 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.562431    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9cc64" event={"ID":"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed","Type":"ContainerStarted","Data":"a30c89391e83a5eec8c37bbb342d5e35bf54b0a6d07a0481d393ac9c6181a022"}
May 20 13:51:21.611377 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:21.611319    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-n9979" podStartSLOduration=1.525777159 podStartE2EDuration="4.611303435s" podCreationTimestamp="2026-05-20 13:51:17 +0000 UTC" firstStartedPulling="2026-05-20 13:51:17.739863053 +0000 UTC m=+44.977479678" lastFinishedPulling="2026-05-20 13:51:20.825389319 +0000 UTC m=+48.063005954" observedRunningTime="2026-05-20 13:51:21.611249204 +0000 UTC m=+48.848865869" watchObservedRunningTime="2026-05-20 13:51:21.611303435 +0000 UTC m=+48.848920080"
May 20 13:51:22.565843 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:22.565809    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d" event={"ID":"c1998859-b3d0-4755-bdb3-cb4ccb164e16","Type":"ContainerStarted","Data":"80276b21c050d0fb29d6d116feb5360df5869253533f2a8a101ae4e83e0dc7a7"}
May 20 13:51:22.566997 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:22.566973    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9cc64" event={"ID":"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed","Type":"ContainerStarted","Data":"feb6562086b05b616d25c5864d0dcb7877161d8a2a1854319b2694e326572f71"}
May 20 13:51:22.651349 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:22.651305    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-579cb6c44-k9mx7" podStartSLOduration=3.364035614 podStartE2EDuration="6.651292239s" podCreationTimestamp="2026-05-20 13:51:16 +0000 UTC" firstStartedPulling="2026-05-20 13:51:17.534209814 +0000 UTC m=+44.771826438" lastFinishedPulling="2026-05-20 13:51:20.821466424 +0000 UTC m=+48.059083063" observedRunningTime="2026-05-20 13:51:21.65587239 +0000 UTC m=+48.893489038" watchObservedRunningTime="2026-05-20 13:51:22.651292239 +0000 UTC m=+49.888908885"
May 20 13:51:23.571410 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:23.571373    2570 generic.go:358] "Generic (PLEG): container finished" podID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerID="4dd59320c3cc38de163ae7f9396721ffbc220ce9ed3324d52d1561e39adff8db" exitCode=0
May 20 13:51:23.571966 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:23.571418    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4cf4bd0-9845-4f94-bf86-154c26adcb33","Type":"ContainerDied","Data":"4dd59320c3cc38de163ae7f9396721ffbc220ce9ed3324d52d1561e39adff8db"}
May 20 13:51:23.573574 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:23.573542    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d" event={"ID":"c1998859-b3d0-4755-bdb3-cb4ccb164e16","Type":"ContainerStarted","Data":"7e3c1c2d561a4279815ebb198492bfa0608c0f7f2e03611d3eb072e023c3e381"}
May 20 13:51:23.573682 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:23.573576    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d" event={"ID":"c1998859-b3d0-4755-bdb3-cb4ccb164e16","Type":"ContainerStarted","Data":"66b58d4ffebb6a7306c3b01540a9b706bb9e4f29a337da31f483417fe04f47f0"}
May 20 13:51:23.574933 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:23.574914    2570 generic.go:358] "Generic (PLEG): container finished" podID="a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed" containerID="feb6562086b05b616d25c5864d0dcb7877161d8a2a1854319b2694e326572f71" exitCode=0
May 20 13:51:23.575041 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:23.574948    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9cc64" event={"ID":"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed","Type":"ContainerDied","Data":"feb6562086b05b616d25c5864d0dcb7877161d8a2a1854319b2694e326572f71"}
May 20 13:51:23.834268 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:23.834150    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-jxt2d" podStartSLOduration=3.67616508 podStartE2EDuration="4.834130508s" podCreationTimestamp="2026-05-20 13:51:19 +0000 UTC" firstStartedPulling="2026-05-20 13:51:21.063732706 +0000 UTC m=+48.301349331" lastFinishedPulling="2026-05-20 13:51:22.221698132 +0000 UTC m=+49.459314759" observedRunningTime="2026-05-20 13:51:23.832996083 +0000 UTC m=+51.070612721" watchObservedRunningTime="2026-05-20 13:51:23.834130508 +0000 UTC m=+51.071747156"
May 20 13:51:23.884551 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:23.884516    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-hnpg6"]
May 20 13:51:23.890901 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:23.890880    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-hnpg6"
May 20 13:51:23.902655 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:23.902635    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
May 20 13:51:23.902758 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:23.902724    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-fqqbv\""
May 20 13:51:23.922292 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:23.922258    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/756758ba-486b-42df-9c73-3f52e08e5da6-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-hnpg6\" (UID: \"756758ba-486b-42df-9c73-3f52e08e5da6\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-hnpg6"
May 20 13:51:23.924317 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:23.924290    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-hnpg6"]
May 20 13:51:24.023363 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:24.023326    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/756758ba-486b-42df-9c73-3f52e08e5da6-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-hnpg6\" (UID: \"756758ba-486b-42df-9c73-3f52e08e5da6\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-hnpg6"
May 20 13:51:24.023570 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:51:24.023474    2570 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
May 20 13:51:24.023570 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:51:24.023566    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/756758ba-486b-42df-9c73-3f52e08e5da6-monitoring-plugin-cert podName:756758ba-486b-42df-9c73-3f52e08e5da6 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:24.52353236 +0000 UTC m=+51.761148985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/756758ba-486b-42df-9c73-3f52e08e5da6-monitoring-plugin-cert") pod "monitoring-plugin-665c9c8584-hnpg6" (UID: "756758ba-486b-42df-9c73-3f52e08e5da6") : secret "monitoring-plugin-cert" not found
May 20 13:51:24.527695 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:24.527660    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/756758ba-486b-42df-9c73-3f52e08e5da6-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-hnpg6\" (UID: \"756758ba-486b-42df-9c73-3f52e08e5da6\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-hnpg6"
May 20 13:51:24.531343 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:24.531313    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/756758ba-486b-42df-9c73-3f52e08e5da6-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-hnpg6\" (UID: \"756758ba-486b-42df-9c73-3f52e08e5da6\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-hnpg6"
May 20 13:51:24.532427 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:24.532407    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-df47p"
May 20 13:51:24.581887 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:24.581843    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9cc64" event={"ID":"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed","Type":"ContainerStarted","Data":"82f36fca4dd50d7897f30b0f61f2041dcf558106900763a7a3afce5b8e546dd5"}
May 20 13:51:24.582351 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:24.581898    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9cc64" event={"ID":"a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed","Type":"ContainerStarted","Data":"e33080282ed3200e987ac30773355e4c7166891943dd5c054177abf709e0eeb4"}
May 20 13:51:24.632901 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:24.632854    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9cc64" podStartSLOduration=4.246523546 podStartE2EDuration="5.632839012s" podCreationTimestamp="2026-05-20 13:51:19 +0000 UTC" firstStartedPulling="2026-05-20 13:51:20.83290443 +0000 UTC m=+48.070521057" lastFinishedPulling="2026-05-20 13:51:22.219219894 +0000 UTC m=+49.456836523" observedRunningTime="2026-05-20 13:51:24.631384111 +0000 UTC m=+51.869000769" watchObservedRunningTime="2026-05-20 13:51:24.632839012 +0000 UTC m=+51.870455672"
May 20 13:51:24.799205 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:24.799111    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-hnpg6"
May 20 13:51:25.221123 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:25.221087    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-hnpg6"]
May 20 13:51:25.224809 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:51:25.224781    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod756758ba_486b_42df_9c73_3f52e08e5da6.slice/crio-6ce9585348388576ee2fc820b5365cf34f07d0c73b6f3015daea196815c9a956 WatchSource:0}: Error finding container 6ce9585348388576ee2fc820b5365cf34f07d0c73b6f3015daea196815c9a956: Status 404 returned error can't find the container with id 6ce9585348388576ee2fc820b5365cf34f07d0c73b6f3015daea196815c9a956
May 20 13:51:25.587464 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:25.587419    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4cf4bd0-9845-4f94-bf86-154c26adcb33","Type":"ContainerStarted","Data":"2fc97db245479b3e647a0e0505604f93b5c1815a6e8091ea9c1078bb768a7768"}
May 20 13:51:25.588774 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:25.588748    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-hnpg6" event={"ID":"756758ba-486b-42df-9c73-3f52e08e5da6","Type":"ContainerStarted","Data":"6ce9585348388576ee2fc820b5365cf34f07d0c73b6f3015daea196815c9a956"}
May 20 13:51:26.597041 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:26.597000    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4cf4bd0-9845-4f94-bf86-154c26adcb33","Type":"ContainerStarted","Data":"a15381c9642bdc8262cf1ff04bd6327fc2f55c96b779bc39d52eec2b9a1d33a4"}
May 20 13:51:26.597410 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:26.597049    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4cf4bd0-9845-4f94-bf86-154c26adcb33","Type":"ContainerStarted","Data":"641dc5613021dd414bdc3fdec8f3f8a78d710e58dff2e6225882d8fba5e064bb"}
May 20 13:51:26.597410 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:26.597063    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4cf4bd0-9845-4f94-bf86-154c26adcb33","Type":"ContainerStarted","Data":"795372ac55f6caa844e8d6ac4540b62cdce29768f6eabe9f07348da7902cb9c4"}
May 20 13:51:26.597410 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:26.597076    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4cf4bd0-9845-4f94-bf86-154c26adcb33","Type":"ContainerStarted","Data":"40faca541e87e7237e424c3bdef70aa06ce9853d5c36691f3f945393bad8a949"}
May 20 13:51:27.356547 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.356513    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:27.356765 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.356560    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:27.362776 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.362754    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:27.488714 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.488684    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-8645b9c94b-9pftl"]
May 20 13:51:27.504153 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.504128    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.523378 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.523352    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8645b9c94b-9pftl"]
May 20 13:51:27.526572 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.526549    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 20 13:51:27.557603 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.557571    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-console-config\") pod \"console-8645b9c94b-9pftl\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") " pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.557780 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.557631    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-trusted-ca-bundle\") pod \"console-8645b9c94b-9pftl\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") " pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.557780 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.557655    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff0f998f-606c-4700-8737-218060a98a69-console-oauth-config\") pod \"console-8645b9c94b-9pftl\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") " pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.557780 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.557706    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-oauth-serving-cert\") pod \"console-8645b9c94b-9pftl\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") " pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.557780 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.557733    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff0f998f-606c-4700-8737-218060a98a69-console-serving-cert\") pod \"console-8645b9c94b-9pftl\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") " pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.557780 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.557748    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrr4j\" (UniqueName: \"kubernetes.io/projected/ff0f998f-606c-4700-8737-218060a98a69-kube-api-access-qrr4j\") pod \"console-8645b9c94b-9pftl\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") " pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.557780 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.557765    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-service-ca\") pod \"console-8645b9c94b-9pftl\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") " pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.612514 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.612489    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:51:27.659065 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.658821    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-console-config\") pod \"console-8645b9c94b-9pftl\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") " pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.659065 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.658880    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-trusted-ca-bundle\") pod \"console-8645b9c94b-9pftl\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") " pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.659065 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.658907    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff0f998f-606c-4700-8737-218060a98a69-console-oauth-config\") pod \"console-8645b9c94b-9pftl\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") " pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.659065 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.658978    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-oauth-serving-cert\") pod \"console-8645b9c94b-9pftl\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") " pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.659065 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.659011    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff0f998f-606c-4700-8737-218060a98a69-console-serving-cert\") pod \"console-8645b9c94b-9pftl\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") " pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.659065 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.659035    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qrr4j\" (UniqueName: \"kubernetes.io/projected/ff0f998f-606c-4700-8737-218060a98a69-kube-api-access-qrr4j\") pod \"console-8645b9c94b-9pftl\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") " pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.659065 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.659065    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-service-ca\") pod \"console-8645b9c94b-9pftl\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") " pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.659644 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.659600    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-console-config\") pod \"console-8645b9c94b-9pftl\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") " pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.659917 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.659892    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-oauth-serving-cert\") pod \"console-8645b9c94b-9pftl\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") " pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.660493 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.660428    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-service-ca\") pod \"console-8645b9c94b-9pftl\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") " pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.660840 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.660761    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-trusted-ca-bundle\") pod \"console-8645b9c94b-9pftl\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") " pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.662718 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.662695    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff0f998f-606c-4700-8737-218060a98a69-console-serving-cert\") pod \"console-8645b9c94b-9pftl\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") " pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.662875 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.662856    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff0f998f-606c-4700-8737-218060a98a69-console-oauth-config\") pod \"console-8645b9c94b-9pftl\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") " pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.687516 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.687486    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrr4j\" (UniqueName: \"kubernetes.io/projected/ff0f998f-606c-4700-8737-218060a98a69-kube-api-access-qrr4j\") pod \"console-8645b9c94b-9pftl\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") " pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.813501 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.813462    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:27.973544 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:27.973518    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8645b9c94b-9pftl"]
May 20 13:51:27.976792 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:51:27.976749    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff0f998f_606c_4700_8737_218060a98a69.slice/crio-9214a6425773c1a3795f3446f7bc0474ba2622054b694feeed6d5b7bd527ad82 WatchSource:0}: Error finding container 9214a6425773c1a3795f3446f7bc0474ba2622054b694feeed6d5b7bd527ad82: Status 404 returned error can't find the container with id 9214a6425773c1a3795f3446f7bc0474ba2622054b694feeed6d5b7bd527ad82
May 20 13:51:28.605783 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:28.605737    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8645b9c94b-9pftl" event={"ID":"ff0f998f-606c-4700-8737-218060a98a69","Type":"ContainerStarted","Data":"a21b750e328839496abcf87f99700f102143398ed47e1e666373aa52858ccccc"}
May 20 13:51:28.605783 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:28.605787    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8645b9c94b-9pftl" event={"ID":"ff0f998f-606c-4700-8737-218060a98a69","Type":"ContainerStarted","Data":"9214a6425773c1a3795f3446f7bc0474ba2622054b694feeed6d5b7bd527ad82"}
May 20 13:51:28.609028 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:28.608999    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4cf4bd0-9845-4f94-bf86-154c26adcb33","Type":"ContainerStarted","Data":"377cb67f2e4c07b941cffb803b24da616f8fde44724533b9c1678211bbcc7786"}
May 20 13:51:28.610481 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:28.610454    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-hnpg6" event={"ID":"756758ba-486b-42df-9c73-3f52e08e5da6","Type":"ContainerStarted","Data":"dc30972000f7ae2e13aeeb9ef8d952d49e96059fb1cda9ee394dfb67335a20da"}
May 20 13:51:28.610852 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:28.610820    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-hnpg6"
May 20 13:51:28.616248 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:28.616225    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-hnpg6"
May 20 13:51:28.717089 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:28.717033    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.516841002 podStartE2EDuration="8.717020304s" podCreationTimestamp="2026-05-20 13:51:20 +0000 UTC" firstStartedPulling="2026-05-20 13:51:21.386705764 +0000 UTC m=+48.624322388" lastFinishedPulling="2026-05-20 13:51:27.586885052 +0000 UTC m=+54.824501690" observedRunningTime="2026-05-20 13:51:28.716928354 +0000 UTC m=+55.954545017" watchObservedRunningTime="2026-05-20 13:51:28.717020304 +0000 UTC m=+55.954636949"
May 20 13:51:28.717379 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:28.717360    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-8645b9c94b-9pftl" podStartSLOduration=1.71735528 podStartE2EDuration="1.71735528s" podCreationTimestamp="2026-05-20 13:51:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:51:28.65170018 +0000 UTC m=+55.889316828" watchObservedRunningTime="2026-05-20 13:51:28.71735528 +0000 UTC m=+55.954971951"
May 20 13:51:28.746166 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:28.746101    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-hnpg6" podStartSLOduration=3.387334523 podStartE2EDuration="5.746080744s" podCreationTimestamp="2026-05-20 13:51:23 +0000 UTC" firstStartedPulling="2026-05-20 13:51:25.226665903 +0000 UTC m=+52.464282531" lastFinishedPulling="2026-05-20 13:51:27.585412127 +0000 UTC m=+54.823028752" observedRunningTime="2026-05-20 13:51:28.745082815 +0000 UTC m=+55.982699484" watchObservedRunningTime="2026-05-20 13:51:28.746080744 +0000 UTC m=+55.983697390"
May 20 13:51:32.497260 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:32.497228    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfhg5"
May 20 13:51:37.814364 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:37.814321    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:37.814770 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:37.814376    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:37.819146 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:37.819123    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:38.642343 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:38.642309    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:51:38.788971 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:38.788938    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-579cb6c44-k9mx7"]
May 20 13:51:39.163838 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:39.163804    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wpd62\" (UniqueName: \"kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62\") pod \"network-check-target-clw2n\" (UID: \"31842022-6a01-49b0-8b26-fb0d882cdf6f\") " pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:51:39.164254 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:39.163875    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs\") pod \"network-metrics-daemon-rhcx7\" (UID: \"6969002d-0b97-4cb6-ae5b-03d5ef9dddf8\") " pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:51:39.172106 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:39.172085    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 20 13:51:39.173567 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:39.173554    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 20 13:51:39.177109 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:39.177081    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6969002d-0b97-4cb6-ae5b-03d5ef9dddf8-metrics-certs\") pod \"network-metrics-daemon-rhcx7\" (UID: \"6969002d-0b97-4cb6-ae5b-03d5ef9dddf8\") " pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:51:39.178746 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:39.178729    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 20 13:51:39.187702 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:39.187675    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpd62\" (UniqueName: \"kubernetes.io/projected/31842022-6a01-49b0-8b26-fb0d882cdf6f-kube-api-access-wpd62\") pod \"network-check-target-clw2n\" (UID: \"31842022-6a01-49b0-8b26-fb0d882cdf6f\") " pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:51:39.277956 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:39.277925    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-68ths\""
May 20 13:51:39.278189 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:39.278172    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qph4s\""
May 20 13:51:39.279175 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:39.279161    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhcx7"
May 20 13:51:39.284685 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:39.284665    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:51:39.447547 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:39.447475    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-clw2n"]
May 20 13:51:39.464520 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:51:39.464483    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31842022_6a01_49b0_8b26_fb0d882cdf6f.slice/crio-5b6e3fdc3ea08ad8c5f1fb124a7b27b30a73485a6753ee841e90fb362b2d07c9 WatchSource:0}: Error finding container 5b6e3fdc3ea08ad8c5f1fb124a7b27b30a73485a6753ee841e90fb362b2d07c9: Status 404 returned error can't find the container with id 5b6e3fdc3ea08ad8c5f1fb124a7b27b30a73485a6753ee841e90fb362b2d07c9
May 20 13:51:39.481023 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:39.480993    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rhcx7"]
May 20 13:51:39.484339 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:51:39.484305    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6969002d_0b97_4cb6_ae5b_03d5ef9dddf8.slice/crio-ac0718f897a3e2aba5eff6257349923f6e29a92e6daf9be2c1d5f009f9cb1dde WatchSource:0}: Error finding container ac0718f897a3e2aba5eff6257349923f6e29a92e6daf9be2c1d5f009f9cb1dde: Status 404 returned error can't find the container with id ac0718f897a3e2aba5eff6257349923f6e29a92e6daf9be2c1d5f009f9cb1dde
May 20 13:51:39.642128 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:39.642083    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-clw2n" event={"ID":"31842022-6a01-49b0-8b26-fb0d882cdf6f","Type":"ContainerStarted","Data":"5b6e3fdc3ea08ad8c5f1fb124a7b27b30a73485a6753ee841e90fb362b2d07c9"}
May 20 13:51:39.642972 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:39.642945    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rhcx7" event={"ID":"6969002d-0b97-4cb6-ae5b-03d5ef9dddf8","Type":"ContainerStarted","Data":"ac0718f897a3e2aba5eff6257349923f6e29a92e6daf9be2c1d5f009f9cb1dde"}
May 20 13:51:40.648166 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:40.648125    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rhcx7" event={"ID":"6969002d-0b97-4cb6-ae5b-03d5ef9dddf8","Type":"ContainerStarted","Data":"8c78612215cbb5368d59d184bf0e42ac390291f5792c67ae3633a903482b2609"}
May 20 13:51:40.648166 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:40.648169    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rhcx7" event={"ID":"6969002d-0b97-4cb6-ae5b-03d5ef9dddf8","Type":"ContainerStarted","Data":"aaba393662a1acaa36f8534ca348377c2e765b55b8e7fd1c6f92b12146e7b66a"}
May 20 13:51:43.424580 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:43.424533    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rhcx7" podStartSLOduration=69.473818393 podStartE2EDuration="1m10.4245147s" podCreationTimestamp="2026-05-20 13:50:33 +0000 UTC" firstStartedPulling="2026-05-20 13:51:39.4864334 +0000 UTC m=+66.724050025" lastFinishedPulling="2026-05-20 13:51:40.437129708 +0000 UTC m=+67.674746332" observedRunningTime="2026-05-20 13:51:40.705116045 +0000 UTC m=+67.942732691" watchObservedRunningTime="2026-05-20 13:51:43.4245147 +0000 UTC m=+70.662131347"
May 20 13:51:43.660569 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:43.660528    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-clw2n" event={"ID":"31842022-6a01-49b0-8b26-fb0d882cdf6f","Type":"ContainerStarted","Data":"7c08c38d06cea93c002cd8c779bea87ec3c381202d45ba42ea463298884dd69c"}
May 20 13:51:43.660770 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:51:43.660651    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:52:03.810195 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:03.810054    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-579cb6c44-k9mx7" podUID="81b01ffe-be78-4c08-8ac6-1a722d9ad441" containerName="console" containerID="cri-o://b17ba85ebad631589b9e1cfe4fec2d2156b45ca5171661f9dc88a5ec109d2b02" gracePeriod=15
May 20 13:52:04.072061 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.072037    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-579cb6c44-k9mx7_81b01ffe-be78-4c08-8ac6-1a722d9ad441/console/0.log"
May 20 13:52:04.072188 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.072123    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:52:04.119693 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.119594    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-clw2n" podStartSLOduration=87.836178945 podStartE2EDuration="1m31.119580006s" podCreationTimestamp="2026-05-20 13:50:33 +0000 UTC" firstStartedPulling="2026-05-20 13:51:39.466586447 +0000 UTC m=+66.704203072" lastFinishedPulling="2026-05-20 13:51:42.749987494 +0000 UTC m=+69.987604133" observedRunningTime="2026-05-20 13:51:43.693551587 +0000 UTC m=+70.931168233" watchObservedRunningTime="2026-05-20 13:52:04.119580006 +0000 UTC m=+91.357196668"
May 20 13:52:04.264634 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.264575    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81b01ffe-be78-4c08-8ac6-1a722d9ad441-service-ca\") pod \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") "
May 20 13:52:04.264634 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.264648    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81b01ffe-be78-4c08-8ac6-1a722d9ad441-console-config\") pod \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") "
May 20 13:52:04.264870 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.264668    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr2wz\" (UniqueName: \"kubernetes.io/projected/81b01ffe-be78-4c08-8ac6-1a722d9ad441-kube-api-access-gr2wz\") pod \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") "
May 20 13:52:04.264870 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.264686    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81b01ffe-be78-4c08-8ac6-1a722d9ad441-console-serving-cert\") pod \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") "
May 20 13:52:04.264870 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.264764    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81b01ffe-be78-4c08-8ac6-1a722d9ad441-oauth-serving-cert\") pod \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") "
May 20 13:52:04.264870 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.264796    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81b01ffe-be78-4c08-8ac6-1a722d9ad441-console-oauth-config\") pod \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\" (UID: \"81b01ffe-be78-4c08-8ac6-1a722d9ad441\") "
May 20 13:52:04.265124 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.265037    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b01ffe-be78-4c08-8ac6-1a722d9ad441-service-ca" (OuterVolumeSpecName: "service-ca") pod "81b01ffe-be78-4c08-8ac6-1a722d9ad441" (UID: "81b01ffe-be78-4c08-8ac6-1a722d9ad441"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:52:04.265199 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.265136    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b01ffe-be78-4c08-8ac6-1a722d9ad441-console-config" (OuterVolumeSpecName: "console-config") pod "81b01ffe-be78-4c08-8ac6-1a722d9ad441" (UID: "81b01ffe-be78-4c08-8ac6-1a722d9ad441"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:52:04.265247 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.265199    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b01ffe-be78-4c08-8ac6-1a722d9ad441-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "81b01ffe-be78-4c08-8ac6-1a722d9ad441" (UID: "81b01ffe-be78-4c08-8ac6-1a722d9ad441"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:52:04.267149 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.267127    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b01ffe-be78-4c08-8ac6-1a722d9ad441-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "81b01ffe-be78-4c08-8ac6-1a722d9ad441" (UID: "81b01ffe-be78-4c08-8ac6-1a722d9ad441"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:52:04.267231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.267166    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b01ffe-be78-4c08-8ac6-1a722d9ad441-kube-api-access-gr2wz" (OuterVolumeSpecName: "kube-api-access-gr2wz") pod "81b01ffe-be78-4c08-8ac6-1a722d9ad441" (UID: "81b01ffe-be78-4c08-8ac6-1a722d9ad441"). InnerVolumeSpecName "kube-api-access-gr2wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:52:04.267274 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.267234    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b01ffe-be78-4c08-8ac6-1a722d9ad441-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "81b01ffe-be78-4c08-8ac6-1a722d9ad441" (UID: "81b01ffe-be78-4c08-8ac6-1a722d9ad441"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:52:04.365467 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.365391    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81b01ffe-be78-4c08-8ac6-1a722d9ad441-oauth-serving-cert\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:52:04.365467 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.365417    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81b01ffe-be78-4c08-8ac6-1a722d9ad441-console-oauth-config\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:52:04.365467 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.365428    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81b01ffe-be78-4c08-8ac6-1a722d9ad441-service-ca\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:52:04.365467 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.365437    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81b01ffe-be78-4c08-8ac6-1a722d9ad441-console-config\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:52:04.365467 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.365445    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr2wz\" (UniqueName: \"kubernetes.io/projected/81b01ffe-be78-4c08-8ac6-1a722d9ad441-kube-api-access-gr2wz\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:52:04.365467 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.365456    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81b01ffe-be78-4c08-8ac6-1a722d9ad441-console-serving-cert\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:52:04.716455 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.716375    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-579cb6c44-k9mx7_81b01ffe-be78-4c08-8ac6-1a722d9ad441/console/0.log"
May 20 13:52:04.716455 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.716421    2570 generic.go:358] "Generic (PLEG): container finished" podID="81b01ffe-be78-4c08-8ac6-1a722d9ad441" containerID="b17ba85ebad631589b9e1cfe4fec2d2156b45ca5171661f9dc88a5ec109d2b02" exitCode=2
May 20 13:52:04.716689 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.716474    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579cb6c44-k9mx7" event={"ID":"81b01ffe-be78-4c08-8ac6-1a722d9ad441","Type":"ContainerDied","Data":"b17ba85ebad631589b9e1cfe4fec2d2156b45ca5171661f9dc88a5ec109d2b02"}
May 20 13:52:04.716689 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.716482    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579cb6c44-k9mx7"
May 20 13:52:04.716689 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.716507    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579cb6c44-k9mx7" event={"ID":"81b01ffe-be78-4c08-8ac6-1a722d9ad441","Type":"ContainerDied","Data":"42677761c74ff7a960fe3ccbf7daf9c8b8b6a93eb813b546d1a1a436ecdc9fd9"}
May 20 13:52:04.716689 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.716526    2570 scope.go:117] "RemoveContainer" containerID="b17ba85ebad631589b9e1cfe4fec2d2156b45ca5171661f9dc88a5ec109d2b02"
May 20 13:52:04.724753 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.724736    2570 scope.go:117] "RemoveContainer" containerID="b17ba85ebad631589b9e1cfe4fec2d2156b45ca5171661f9dc88a5ec109d2b02"
May 20 13:52:04.725027 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:04.725008    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b17ba85ebad631589b9e1cfe4fec2d2156b45ca5171661f9dc88a5ec109d2b02\": container with ID starting with b17ba85ebad631589b9e1cfe4fec2d2156b45ca5171661f9dc88a5ec109d2b02 not found: ID does not exist" containerID="b17ba85ebad631589b9e1cfe4fec2d2156b45ca5171661f9dc88a5ec109d2b02"
May 20 13:52:04.725095 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.725040    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b17ba85ebad631589b9e1cfe4fec2d2156b45ca5171661f9dc88a5ec109d2b02"} err="failed to get container status \"b17ba85ebad631589b9e1cfe4fec2d2156b45ca5171661f9dc88a5ec109d2b02\": rpc error: code = NotFound desc = could not find container \"b17ba85ebad631589b9e1cfe4fec2d2156b45ca5171661f9dc88a5ec109d2b02\": container with ID starting with b17ba85ebad631589b9e1cfe4fec2d2156b45ca5171661f9dc88a5ec109d2b02 not found: ID does not exist"
May 20 13:52:04.748495 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.748470    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-579cb6c44-k9mx7"]
May 20 13:52:04.758649 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:04.758597    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-579cb6c44-k9mx7"]
May 20 13:52:05.363345 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:05.363299    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81b01ffe-be78-4c08-8ac6-1a722d9ad441" path="/var/lib/kubelet/pods/81b01ffe-be78-4c08-8ac6-1a722d9ad441/volumes"
May 20 13:52:14.667606 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:14.667573    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-clw2n"
May 20 13:52:44.424823 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:44.424774    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:52:44.425584 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:44.425402    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="alertmanager" containerID="cri-o://2fc97db245479b3e647a0e0505604f93b5c1815a6e8091ea9c1078bb768a7768" gracePeriod=120
May 20 13:52:44.425584 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:44.425432    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="kube-rbac-proxy-metric" containerID="cri-o://a15381c9642bdc8262cf1ff04bd6327fc2f55c96b779bc39d52eec2b9a1d33a4" gracePeriod=120
May 20 13:52:44.425584 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:44.425460    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="config-reloader" containerID="cri-o://40faca541e87e7237e424c3bdef70aa06ce9853d5c36691f3f945393bad8a949" gracePeriod=120
May 20 13:52:44.425584 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:44.425446    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="kube-rbac-proxy-web" containerID="cri-o://795372ac55f6caa844e8d6ac4540b62cdce29768f6eabe9f07348da7902cb9c4" gracePeriod=120
May 20 13:52:44.425584 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:44.425489    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="prom-label-proxy" containerID="cri-o://377cb67f2e4c07b941cffb803b24da616f8fde44724533b9c1678211bbcc7786" gracePeriod=120
May 20 13:52:44.426507 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:44.425502    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="kube-rbac-proxy" containerID="cri-o://641dc5613021dd414bdc3fdec8f3f8a78d710e58dff2e6225882d8fba5e064bb" gracePeriod=120
May 20 13:52:44.822313 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:44.822279    2570 generic.go:358] "Generic (PLEG): container finished" podID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerID="377cb67f2e4c07b941cffb803b24da616f8fde44724533b9c1678211bbcc7786" exitCode=0
May 20 13:52:44.822313 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:44.822305    2570 generic.go:358] "Generic (PLEG): container finished" podID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerID="641dc5613021dd414bdc3fdec8f3f8a78d710e58dff2e6225882d8fba5e064bb" exitCode=0
May 20 13:52:44.822313 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:44.822312    2570 generic.go:358] "Generic (PLEG): container finished" podID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerID="40faca541e87e7237e424c3bdef70aa06ce9853d5c36691f3f945393bad8a949" exitCode=0
May 20 13:52:44.822313 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:44.822317    2570 generic.go:358] "Generic (PLEG): container finished" podID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerID="2fc97db245479b3e647a0e0505604f93b5c1815a6e8091ea9c1078bb768a7768" exitCode=0
May 20 13:52:44.822562 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:44.822339    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4cf4bd0-9845-4f94-bf86-154c26adcb33","Type":"ContainerDied","Data":"377cb67f2e4c07b941cffb803b24da616f8fde44724533b9c1678211bbcc7786"}
May 20 13:52:44.822562 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:44.822361    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4cf4bd0-9845-4f94-bf86-154c26adcb33","Type":"ContainerDied","Data":"641dc5613021dd414bdc3fdec8f3f8a78d710e58dff2e6225882d8fba5e064bb"}
May 20 13:52:44.822562 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:44.822372    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4cf4bd0-9845-4f94-bf86-154c26adcb33","Type":"ContainerDied","Data":"40faca541e87e7237e424c3bdef70aa06ce9853d5c36691f3f945393bad8a949"}
May 20 13:52:44.822562 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:44.822380    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4cf4bd0-9845-4f94-bf86-154c26adcb33","Type":"ContainerDied","Data":"2fc97db245479b3e647a0e0505604f93b5c1815a6e8091ea9c1078bb768a7768"}
May 20 13:52:45.689192 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.689169    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:45.779231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.779197    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4cf4bd0-9845-4f94-bf86-154c26adcb33-tls-assets\") pod \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") "
May 20 13:52:45.779410 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.779238    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nct2z\" (UniqueName: \"kubernetes.io/projected/c4cf4bd0-9845-4f94-bf86-154c26adcb33-kube-api-access-nct2z\") pod \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") "
May 20 13:52:45.779410 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.779273    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-config-volume\") pod \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") "
May 20 13:52:45.779410 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.779298    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") "
May 20 13:52:45.779410 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.779329    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4cf4bd0-9845-4f94-bf86-154c26adcb33-metrics-client-ca\") pod \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") "
May 20 13:52:45.779410 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.779351    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4cf4bd0-9845-4f94-bf86-154c26adcb33-alertmanager-trusted-ca-bundle\") pod \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") "
May 20 13:52:45.779410 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.779382    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4cf4bd0-9845-4f94-bf86-154c26adcb33-config-out\") pod \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") "
May 20 13:52:45.779410 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.779407    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-kube-rbac-proxy-web\") pod \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") "
May 20 13:52:45.779803 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.779458    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-kube-rbac-proxy\") pod \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") "
May 20 13:52:45.779803 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.779488    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-cluster-tls-config\") pod \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") "
May 20 13:52:45.779803 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.779527    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-web-config\") pod \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") "
May 20 13:52:45.779803 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.779571    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c4cf4bd0-9845-4f94-bf86-154c26adcb33-alertmanager-main-db\") pod \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") "
May 20 13:52:45.779803 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.779607    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-main-tls\") pod \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\" (UID: \"c4cf4bd0-9845-4f94-bf86-154c26adcb33\") "
May 20 13:52:45.779803 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.779773    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4cf4bd0-9845-4f94-bf86-154c26adcb33-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c4cf4bd0-9845-4f94-bf86-154c26adcb33" (UID: "c4cf4bd0-9845-4f94-bf86-154c26adcb33"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:52:45.780100 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.779920    2570 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4cf4bd0-9845-4f94-bf86-154c26adcb33-metrics-client-ca\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:52:45.780512 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.780428    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4cf4bd0-9845-4f94-bf86-154c26adcb33-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c4cf4bd0-9845-4f94-bf86-154c26adcb33" (UID: "c4cf4bd0-9845-4f94-bf86-154c26adcb33"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:52:45.781188 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.781156    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4cf4bd0-9845-4f94-bf86-154c26adcb33-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c4cf4bd0-9845-4f94-bf86-154c26adcb33" (UID: "c4cf4bd0-9845-4f94-bf86-154c26adcb33"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 13:52:45.782412 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.782365    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c4cf4bd0-9845-4f94-bf86-154c26adcb33" (UID: "c4cf4bd0-9845-4f94-bf86-154c26adcb33"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:52:45.782768 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.782743    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4cf4bd0-9845-4f94-bf86-154c26adcb33-kube-api-access-nct2z" (OuterVolumeSpecName: "kube-api-access-nct2z") pod "c4cf4bd0-9845-4f94-bf86-154c26adcb33" (UID: "c4cf4bd0-9845-4f94-bf86-154c26adcb33"). InnerVolumeSpecName "kube-api-access-nct2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:52:45.783537 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.783508    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4cf4bd0-9845-4f94-bf86-154c26adcb33-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c4cf4bd0-9845-4f94-bf86-154c26adcb33" (UID: "c4cf4bd0-9845-4f94-bf86-154c26adcb33"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:52:45.783945 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.783885    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c4cf4bd0-9845-4f94-bf86-154c26adcb33" (UID: "c4cf4bd0-9845-4f94-bf86-154c26adcb33"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:52:45.784314 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.784289    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c4cf4bd0-9845-4f94-bf86-154c26adcb33" (UID: "c4cf4bd0-9845-4f94-bf86-154c26adcb33"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:52:45.784399 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.784375    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-config-volume" (OuterVolumeSpecName: "config-volume") pod "c4cf4bd0-9845-4f94-bf86-154c26adcb33" (UID: "c4cf4bd0-9845-4f94-bf86-154c26adcb33"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:52:45.784948 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.784901    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c4cf4bd0-9845-4f94-bf86-154c26adcb33" (UID: "c4cf4bd0-9845-4f94-bf86-154c26adcb33"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:52:45.785048 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.784990    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4cf4bd0-9845-4f94-bf86-154c26adcb33-config-out" (OuterVolumeSpecName: "config-out") pod "c4cf4bd0-9845-4f94-bf86-154c26adcb33" (UID: "c4cf4bd0-9845-4f94-bf86-154c26adcb33"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 13:52:45.787568 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.787546    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c4cf4bd0-9845-4f94-bf86-154c26adcb33" (UID: "c4cf4bd0-9845-4f94-bf86-154c26adcb33"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:52:45.795213 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.795188    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-web-config" (OuterVolumeSpecName: "web-config") pod "c4cf4bd0-9845-4f94-bf86-154c26adcb33" (UID: "c4cf4bd0-9845-4f94-bf86-154c26adcb33"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:52:45.828296 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.828264    2570 generic.go:358] "Generic (PLEG): container finished" podID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerID="a15381c9642bdc8262cf1ff04bd6327fc2f55c96b779bc39d52eec2b9a1d33a4" exitCode=0
May 20 13:52:45.828296 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.828294    2570 generic.go:358] "Generic (PLEG): container finished" podID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerID="795372ac55f6caa844e8d6ac4540b62cdce29768f6eabe9f07348da7902cb9c4" exitCode=0
May 20 13:52:45.828458 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.828346    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4cf4bd0-9845-4f94-bf86-154c26adcb33","Type":"ContainerDied","Data":"a15381c9642bdc8262cf1ff04bd6327fc2f55c96b779bc39d52eec2b9a1d33a4"}
May 20 13:52:45.828458 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.828386    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4cf4bd0-9845-4f94-bf86-154c26adcb33","Type":"ContainerDied","Data":"795372ac55f6caa844e8d6ac4540b62cdce29768f6eabe9f07348da7902cb9c4"}
May 20 13:52:45.828458 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.828399    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4cf4bd0-9845-4f94-bf86-154c26adcb33","Type":"ContainerDied","Data":"6f3fc791656ca153ff8b31fc25c0d0bb958b6dd70aa370cb9b098674f9b03f32"}
May 20 13:52:45.828458 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.828401    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:45.828458 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.828451    2570 scope.go:117] "RemoveContainer" containerID="377cb67f2e4c07b941cffb803b24da616f8fde44724533b9c1678211bbcc7786"
May 20 13:52:45.835555 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.835537    2570 scope.go:117] "RemoveContainer" containerID="a15381c9642bdc8262cf1ff04bd6327fc2f55c96b779bc39d52eec2b9a1d33a4"
May 20 13:52:45.841944 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.841929    2570 scope.go:117] "RemoveContainer" containerID="641dc5613021dd414bdc3fdec8f3f8a78d710e58dff2e6225882d8fba5e064bb"
May 20 13:52:45.847948 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.847933    2570 scope.go:117] "RemoveContainer" containerID="795372ac55f6caa844e8d6ac4540b62cdce29768f6eabe9f07348da7902cb9c4"
May 20 13:52:45.853895 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.853878    2570 scope.go:117] "RemoveContainer" containerID="40faca541e87e7237e424c3bdef70aa06ce9853d5c36691f3f945393bad8a949"
May 20 13:52:45.859989 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.859970    2570 scope.go:117] "RemoveContainer" containerID="2fc97db245479b3e647a0e0505604f93b5c1815a6e8091ea9c1078bb768a7768"
May 20 13:52:45.864256 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.864230    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:52:45.866542 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.866520    2570 scope.go:117] "RemoveContainer" containerID="4dd59320c3cc38de163ae7f9396721ffbc220ce9ed3324d52d1561e39adff8db"
May 20 13:52:45.872827 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.872797    2570 scope.go:117] "RemoveContainer" containerID="377cb67f2e4c07b941cffb803b24da616f8fde44724533b9c1678211bbcc7786"
May 20 13:52:45.873071 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:45.873053    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"377cb67f2e4c07b941cffb803b24da616f8fde44724533b9c1678211bbcc7786\": container with ID starting with 377cb67f2e4c07b941cffb803b24da616f8fde44724533b9c1678211bbcc7786 not found: ID does not exist" containerID="377cb67f2e4c07b941cffb803b24da616f8fde44724533b9c1678211bbcc7786"
May 20 13:52:45.873112 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.873084    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377cb67f2e4c07b941cffb803b24da616f8fde44724533b9c1678211bbcc7786"} err="failed to get container status \"377cb67f2e4c07b941cffb803b24da616f8fde44724533b9c1678211bbcc7786\": rpc error: code = NotFound desc = could not find container \"377cb67f2e4c07b941cffb803b24da616f8fde44724533b9c1678211bbcc7786\": container with ID starting with 377cb67f2e4c07b941cffb803b24da616f8fde44724533b9c1678211bbcc7786 not found: ID does not exist"
May 20 13:52:45.873112 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.873106    2570 scope.go:117] "RemoveContainer" containerID="a15381c9642bdc8262cf1ff04bd6327fc2f55c96b779bc39d52eec2b9a1d33a4"
May 20 13:52:45.873335 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:45.873315    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a15381c9642bdc8262cf1ff04bd6327fc2f55c96b779bc39d52eec2b9a1d33a4\": container with ID starting with a15381c9642bdc8262cf1ff04bd6327fc2f55c96b779bc39d52eec2b9a1d33a4 not found: ID does not exist" containerID="a15381c9642bdc8262cf1ff04bd6327fc2f55c96b779bc39d52eec2b9a1d33a4"
May 20 13:52:45.873377 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.873342    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15381c9642bdc8262cf1ff04bd6327fc2f55c96b779bc39d52eec2b9a1d33a4"} err="failed to get container status \"a15381c9642bdc8262cf1ff04bd6327fc2f55c96b779bc39d52eec2b9a1d33a4\": rpc error: code = NotFound desc = could not find container \"a15381c9642bdc8262cf1ff04bd6327fc2f55c96b779bc39d52eec2b9a1d33a4\": container with ID starting with a15381c9642bdc8262cf1ff04bd6327fc2f55c96b779bc39d52eec2b9a1d33a4 not found: ID does not exist"
May 20 13:52:45.873377 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.873359    2570 scope.go:117] "RemoveContainer" containerID="641dc5613021dd414bdc3fdec8f3f8a78d710e58dff2e6225882d8fba5e064bb"
May 20 13:52:45.873583 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:45.873567    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"641dc5613021dd414bdc3fdec8f3f8a78d710e58dff2e6225882d8fba5e064bb\": container with ID starting with 641dc5613021dd414bdc3fdec8f3f8a78d710e58dff2e6225882d8fba5e064bb not found: ID does not exist" containerID="641dc5613021dd414bdc3fdec8f3f8a78d710e58dff2e6225882d8fba5e064bb"
May 20 13:52:45.873658 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.873584    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"641dc5613021dd414bdc3fdec8f3f8a78d710e58dff2e6225882d8fba5e064bb"} err="failed to get container status \"641dc5613021dd414bdc3fdec8f3f8a78d710e58dff2e6225882d8fba5e064bb\": rpc error: code = NotFound desc = could not find container \"641dc5613021dd414bdc3fdec8f3f8a78d710e58dff2e6225882d8fba5e064bb\": container with ID starting with 641dc5613021dd414bdc3fdec8f3f8a78d710e58dff2e6225882d8fba5e064bb not found: ID does not exist"
May 20 13:52:45.873658 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.873596    2570 scope.go:117] "RemoveContainer" containerID="795372ac55f6caa844e8d6ac4540b62cdce29768f6eabe9f07348da7902cb9c4"
May 20 13:52:45.873822 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:45.873806    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795372ac55f6caa844e8d6ac4540b62cdce29768f6eabe9f07348da7902cb9c4\": container with ID starting with 795372ac55f6caa844e8d6ac4540b62cdce29768f6eabe9f07348da7902cb9c4 not found: ID does not exist" containerID="795372ac55f6caa844e8d6ac4540b62cdce29768f6eabe9f07348da7902cb9c4"
May 20 13:52:45.873864 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.873828    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795372ac55f6caa844e8d6ac4540b62cdce29768f6eabe9f07348da7902cb9c4"} err="failed to get container status \"795372ac55f6caa844e8d6ac4540b62cdce29768f6eabe9f07348da7902cb9c4\": rpc error: code = NotFound desc = could not find container \"795372ac55f6caa844e8d6ac4540b62cdce29768f6eabe9f07348da7902cb9c4\": container with ID starting with 795372ac55f6caa844e8d6ac4540b62cdce29768f6eabe9f07348da7902cb9c4 not found: ID does not exist"
May 20 13:52:45.873864 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.873846    2570 scope.go:117] "RemoveContainer" containerID="40faca541e87e7237e424c3bdef70aa06ce9853d5c36691f3f945393bad8a949"
May 20 13:52:45.874066 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:45.874046    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40faca541e87e7237e424c3bdef70aa06ce9853d5c36691f3f945393bad8a949\": container with ID starting with 40faca541e87e7237e424c3bdef70aa06ce9853d5c36691f3f945393bad8a949 not found: ID does not exist" containerID="40faca541e87e7237e424c3bdef70aa06ce9853d5c36691f3f945393bad8a949"
May 20 13:52:45.874143 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.874068    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40faca541e87e7237e424c3bdef70aa06ce9853d5c36691f3f945393bad8a949"} err="failed to get container status \"40faca541e87e7237e424c3bdef70aa06ce9853d5c36691f3f945393bad8a949\": rpc error: code = NotFound desc = could not find container \"40faca541e87e7237e424c3bdef70aa06ce9853d5c36691f3f945393bad8a949\": container with ID starting with 40faca541e87e7237e424c3bdef70aa06ce9853d5c36691f3f945393bad8a949 not found: ID does not exist"
May 20 13:52:45.874143 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.874083    2570 scope.go:117] "RemoveContainer" containerID="2fc97db245479b3e647a0e0505604f93b5c1815a6e8091ea9c1078bb768a7768"
May 20 13:52:45.874416 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:45.874395    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc97db245479b3e647a0e0505604f93b5c1815a6e8091ea9c1078bb768a7768\": container with ID starting with 2fc97db245479b3e647a0e0505604f93b5c1815a6e8091ea9c1078bb768a7768 not found: ID does not exist" containerID="2fc97db245479b3e647a0e0505604f93b5c1815a6e8091ea9c1078bb768a7768"
May 20 13:52:45.874474 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.874422    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc97db245479b3e647a0e0505604f93b5c1815a6e8091ea9c1078bb768a7768"} err="failed to get container status \"2fc97db245479b3e647a0e0505604f93b5c1815a6e8091ea9c1078bb768a7768\": rpc error: code = NotFound desc = could not find container \"2fc97db245479b3e647a0e0505604f93b5c1815a6e8091ea9c1078bb768a7768\": container with ID starting with 2fc97db245479b3e647a0e0505604f93b5c1815a6e8091ea9c1078bb768a7768 not found: ID does not exist"
May 20 13:52:45.874474 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.874436    2570 scope.go:117] "RemoveContainer" containerID="4dd59320c3cc38de163ae7f9396721ffbc220ce9ed3324d52d1561e39adff8db"
May 20 13:52:45.874679 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:45.874661    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dd59320c3cc38de163ae7f9396721ffbc220ce9ed3324d52d1561e39adff8db\": container with ID starting with 4dd59320c3cc38de163ae7f9396721ffbc220ce9ed3324d52d1561e39adff8db not found: ID does not exist" containerID="4dd59320c3cc38de163ae7f9396721ffbc220ce9ed3324d52d1561e39adff8db"
May 20 13:52:45.874742 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.874683    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd59320c3cc38de163ae7f9396721ffbc220ce9ed3324d52d1561e39adff8db"} err="failed to get container status \"4dd59320c3cc38de163ae7f9396721ffbc220ce9ed3324d52d1561e39adff8db\": rpc error: code = NotFound desc = could not find container \"4dd59320c3cc38de163ae7f9396721ffbc220ce9ed3324d52d1561e39adff8db\": container with ID starting with 4dd59320c3cc38de163ae7f9396721ffbc220ce9ed3324d52d1561e39adff8db not found: ID does not exist"
May 20 13:52:45.874742 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.874695    2570 scope.go:117] "RemoveContainer" containerID="377cb67f2e4c07b941cffb803b24da616f8fde44724533b9c1678211bbcc7786"
May 20 13:52:45.874920 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.874894    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377cb67f2e4c07b941cffb803b24da616f8fde44724533b9c1678211bbcc7786"} err="failed to get container status \"377cb67f2e4c07b941cffb803b24da616f8fde44724533b9c1678211bbcc7786\": rpc error: code = NotFound desc = could not find container \"377cb67f2e4c07b941cffb803b24da616f8fde44724533b9c1678211bbcc7786\": container with ID starting with 377cb67f2e4c07b941cffb803b24da616f8fde44724533b9c1678211bbcc7786 not found: ID does not exist"
May 20 13:52:45.874962 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.874921    2570 scope.go:117] "RemoveContainer" containerID="a15381c9642bdc8262cf1ff04bd6327fc2f55c96b779bc39d52eec2b9a1d33a4"
May 20 13:52:45.875138 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.875120    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15381c9642bdc8262cf1ff04bd6327fc2f55c96b779bc39d52eec2b9a1d33a4"} err="failed to get container status \"a15381c9642bdc8262cf1ff04bd6327fc2f55c96b779bc39d52eec2b9a1d33a4\": rpc error: code = NotFound desc = could not find container \"a15381c9642bdc8262cf1ff04bd6327fc2f55c96b779bc39d52eec2b9a1d33a4\": container with ID starting with a15381c9642bdc8262cf1ff04bd6327fc2f55c96b779bc39d52eec2b9a1d33a4 not found: ID does not exist"
May 20 13:52:45.875188 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.875139    2570 scope.go:117] "RemoveContainer" containerID="641dc5613021dd414bdc3fdec8f3f8a78d710e58dff2e6225882d8fba5e064bb"
May 20 13:52:45.875355 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.875339    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"641dc5613021dd414bdc3fdec8f3f8a78d710e58dff2e6225882d8fba5e064bb"} err="failed to get container status \"641dc5613021dd414bdc3fdec8f3f8a78d710e58dff2e6225882d8fba5e064bb\": rpc error: code = NotFound desc = could not find container \"641dc5613021dd414bdc3fdec8f3f8a78d710e58dff2e6225882d8fba5e064bb\": container with ID starting with 641dc5613021dd414bdc3fdec8f3f8a78d710e58dff2e6225882d8fba5e064bb not found: ID does not exist"
May 20 13:52:45.875404 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.875356    2570 scope.go:117] "RemoveContainer" containerID="795372ac55f6caa844e8d6ac4540b62cdce29768f6eabe9f07348da7902cb9c4"
May 20 13:52:45.875546 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.875530    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795372ac55f6caa844e8d6ac4540b62cdce29768f6eabe9f07348da7902cb9c4"} err="failed to get container status \"795372ac55f6caa844e8d6ac4540b62cdce29768f6eabe9f07348da7902cb9c4\": rpc error: code = NotFound desc = could not find container \"795372ac55f6caa844e8d6ac4540b62cdce29768f6eabe9f07348da7902cb9c4\": container with ID starting with 795372ac55f6caa844e8d6ac4540b62cdce29768f6eabe9f07348da7902cb9c4 not found: ID does not exist"
May 20 13:52:45.875687 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.875547    2570 scope.go:117] "RemoveContainer" containerID="40faca541e87e7237e424c3bdef70aa06ce9853d5c36691f3f945393bad8a949"
May 20 13:52:45.875868 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.875842    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40faca541e87e7237e424c3bdef70aa06ce9853d5c36691f3f945393bad8a949"} err="failed to get container status \"40faca541e87e7237e424c3bdef70aa06ce9853d5c36691f3f945393bad8a949\": rpc error: code = NotFound desc = could not find container \"40faca541e87e7237e424c3bdef70aa06ce9853d5c36691f3f945393bad8a949\": container with ID starting with 40faca541e87e7237e424c3bdef70aa06ce9853d5c36691f3f945393bad8a949 not found: ID does not exist"
May 20 13:52:45.875916 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.875870    2570 scope.go:117] "RemoveContainer" containerID="2fc97db245479b3e647a0e0505604f93b5c1815a6e8091ea9c1078bb768a7768"
May 20 13:52:45.876073 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.876054    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc97db245479b3e647a0e0505604f93b5c1815a6e8091ea9c1078bb768a7768"} err="failed to get container status \"2fc97db245479b3e647a0e0505604f93b5c1815a6e8091ea9c1078bb768a7768\": rpc error: code = NotFound desc = could not find container \"2fc97db245479b3e647a0e0505604f93b5c1815a6e8091ea9c1078bb768a7768\": container with ID starting with 2fc97db245479b3e647a0e0505604f93b5c1815a6e8091ea9c1078bb768a7768 not found: ID does not exist"
May 20 13:52:45.876125 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.876074    2570 scope.go:117] "RemoveContainer" containerID="4dd59320c3cc38de163ae7f9396721ffbc220ce9ed3324d52d1561e39adff8db"
May 20 13:52:45.876276 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.876261    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd59320c3cc38de163ae7f9396721ffbc220ce9ed3324d52d1561e39adff8db"} err="failed to get container status \"4dd59320c3cc38de163ae7f9396721ffbc220ce9ed3324d52d1561e39adff8db\": rpc error: code = NotFound desc = could not find container \"4dd59320c3cc38de163ae7f9396721ffbc220ce9ed3324d52d1561e39adff8db\": container with ID starting with 4dd59320c3cc38de163ae7f9396721ffbc220ce9ed3324d52d1561e39adff8db not found: ID does not exist"
May 20 13:52:45.880857 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.880831    2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:52:45.880976 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.880863    2570 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-cluster-tls-config\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:52:45.880976 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.880879    2570 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-web-config\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:52:45.880976 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.880893    2570 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c4cf4bd0-9845-4f94-bf86-154c26adcb33-alertmanager-main-db\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:52:45.880976 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.880907    2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-main-tls\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:52:45.880976 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.880919    2570 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4cf4bd0-9845-4f94-bf86-154c26adcb33-tls-assets\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:52:45.880976 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.880933    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nct2z\" (UniqueName: \"kubernetes.io/projected/c4cf4bd0-9845-4f94-bf86-154c26adcb33-kube-api-access-nct2z\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:52:45.880976 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.880944    2570 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-config-volume\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:52:45.880976 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.880954    2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:52:45.880976 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.880964    2570 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4cf4bd0-9845-4f94-bf86-154c26adcb33-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:52:45.880976 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.880972    2570 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4cf4bd0-9845-4f94-bf86-154c26adcb33-config-out\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:52:45.880976 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.880980    2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4cf4bd0-9845-4f94-bf86-154c26adcb33-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:52:45.882408 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.882390    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:52:45.943797 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.943761    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:52:45.944016 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944002    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="alertmanager"
May 20 13:52:45.944079 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944017    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="alertmanager"
May 20 13:52:45.944079 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944032    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="config-reloader"
May 20 13:52:45.944079 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944038    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="config-reloader"
May 20 13:52:45.944079 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944045    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="kube-rbac-proxy"
May 20 13:52:45.944079 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944050    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="kube-rbac-proxy"
May 20 13:52:45.944079 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944058    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b01ffe-be78-4c08-8ac6-1a722d9ad441" containerName="console"
May 20 13:52:45.944079 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944065    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b01ffe-be78-4c08-8ac6-1a722d9ad441" containerName="console"
May 20 13:52:45.944079 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944070    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="kube-rbac-proxy-metric"
May 20 13:52:45.944079 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944075    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="kube-rbac-proxy-metric"
May 20 13:52:45.944079 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944081    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="prom-label-proxy"
May 20 13:52:45.944079 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944086    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="prom-label-proxy"
May 20 13:52:45.944566 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944093    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="init-config-reloader"
May 20 13:52:45.944566 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944098    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="init-config-reloader"
May 20 13:52:45.944566 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944104    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="kube-rbac-proxy-web"
May 20 13:52:45.944566 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944110    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="kube-rbac-proxy-web"
May 20 13:52:45.944566 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944147    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="81b01ffe-be78-4c08-8ac6-1a722d9ad441" containerName="console"
May 20 13:52:45.944566 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944154    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="kube-rbac-proxy"
May 20 13:52:45.944566 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944161    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="kube-rbac-proxy-web"
May 20 13:52:45.944566 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944167    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="prom-label-proxy"
May 20 13:52:45.944566 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944173    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="alertmanager"
May 20 13:52:45.944566 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944178    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="config-reloader"
May 20 13:52:45.944566 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.944184    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" containerName="kube-rbac-proxy-metric"
May 20 13:52:45.948954 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.948938    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:45.953774 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.953748    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 20 13:52:45.953774 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.953763    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 20 13:52:45.953959 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.953946    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 20 13:52:45.954084 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.954068    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 20 13:52:45.954570 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.954552    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 20 13:52:45.955150 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.955117    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 20 13:52:45.955150 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.955128    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 20 13:52:45.955303 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.955158    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tclzk\""
May 20 13:52:45.955636 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.955602    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 20 13:52:45.964456 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.964434    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 20 13:52:45.994112 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:45.994081    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:52:46.081973 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.081893    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0807f6a1-c649-418f-acde-1ea45476efe2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.081973 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.081932    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0807f6a1-c649-418f-acde-1ea45476efe2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.081973 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.081963    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0807f6a1-c649-418f-acde-1ea45476efe2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.082182 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.081990    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0807f6a1-c649-418f-acde-1ea45476efe2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.082182 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.082012    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0807f6a1-c649-418f-acde-1ea45476efe2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.082182 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.082031    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0807f6a1-c649-418f-acde-1ea45476efe2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.082182 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.082062    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg8jr\" (UniqueName: \"kubernetes.io/projected/0807f6a1-c649-418f-acde-1ea45476efe2-kube-api-access-lg8jr\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.082182 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.082092    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0807f6a1-c649-418f-acde-1ea45476efe2-config-volume\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.082182 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.082126    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0807f6a1-c649-418f-acde-1ea45476efe2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.082182 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.082144    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0807f6a1-c649-418f-acde-1ea45476efe2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.082182 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.082157    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0807f6a1-c649-418f-acde-1ea45476efe2-web-config\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.082182 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.082173    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0807f6a1-c649-418f-acde-1ea45476efe2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.082441 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.082212    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0807f6a1-c649-418f-acde-1ea45476efe2-config-out\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.183914 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.183862    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0807f6a1-c649-418f-acde-1ea45476efe2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.184091 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.183940    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0807f6a1-c649-418f-acde-1ea45476efe2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.184091 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.183980    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0807f6a1-c649-418f-acde-1ea45476efe2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.184091 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.184011    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0807f6a1-c649-418f-acde-1ea45476efe2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.184091 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.184053    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0807f6a1-c649-418f-acde-1ea45476efe2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.184091 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.184088    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lg8jr\" (UniqueName: \"kubernetes.io/projected/0807f6a1-c649-418f-acde-1ea45476efe2-kube-api-access-lg8jr\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.184330 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.184117    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0807f6a1-c649-418f-acde-1ea45476efe2-config-volume\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.184330 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.184173    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0807f6a1-c649-418f-acde-1ea45476efe2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.184330 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.184214    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0807f6a1-c649-418f-acde-1ea45476efe2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.184330 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.184243    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0807f6a1-c649-418f-acde-1ea45476efe2-web-config\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.184330 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.184276    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0807f6a1-c649-418f-acde-1ea45476efe2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.184330 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.184312    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0807f6a1-c649-418f-acde-1ea45476efe2-config-out\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.184571 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.184360    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0807f6a1-c649-418f-acde-1ea45476efe2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.185231 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.185204    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0807f6a1-c649-418f-acde-1ea45476efe2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.188463 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.188410    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0807f6a1-c649-418f-acde-1ea45476efe2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.188769 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.188741    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0807f6a1-c649-418f-acde-1ea45476efe2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.189943 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.189918    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0807f6a1-c649-418f-acde-1ea45476efe2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.190380 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.190354    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0807f6a1-c649-418f-acde-1ea45476efe2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.190481 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.190458    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0807f6a1-c649-418f-acde-1ea45476efe2-config-out\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.190544 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.190487    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0807f6a1-c649-418f-acde-1ea45476efe2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.191088 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.191064    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0807f6a1-c649-418f-acde-1ea45476efe2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.191600 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.191575    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0807f6a1-c649-418f-acde-1ea45476efe2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.192023 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.192003    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0807f6a1-c649-418f-acde-1ea45476efe2-config-volume\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.192111 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.192094    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0807f6a1-c649-418f-acde-1ea45476efe2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.192203 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.192188    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0807f6a1-c649-418f-acde-1ea45476efe2-web-config\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.198854 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.198837    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg8jr\" (UniqueName: \"kubernetes.io/projected/0807f6a1-c649-418f-acde-1ea45476efe2-kube-api-access-lg8jr\") pod \"alertmanager-main-0\" (UID: \"0807f6a1-c649-418f-acde-1ea45476efe2\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.258239 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.258214    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:46.405699 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.405655    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:52:46.411172 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:52:46.411147    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0807f6a1_c649_418f_acde_1ea45476efe2.slice/crio-6957533dcb59613c8bedc212d2ab1646378e64530131ee3e590fb58323a262d8 WatchSource:0}: Error finding container 6957533dcb59613c8bedc212d2ab1646378e64530131ee3e590fb58323a262d8: Status 404 returned error can't find the container with id 6957533dcb59613c8bedc212d2ab1646378e64530131ee3e590fb58323a262d8
May 20 13:52:46.834566 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.834532    2570 generic.go:358] "Generic (PLEG): container finished" podID="0807f6a1-c649-418f-acde-1ea45476efe2" containerID="cd1f62301fa89b143bcb8a2b215070fad9f7f49a2e289729ed03fe6c9dbd17ee" exitCode=0
May 20 13:52:46.835054 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.834642    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0807f6a1-c649-418f-acde-1ea45476efe2","Type":"ContainerDied","Data":"cd1f62301fa89b143bcb8a2b215070fad9f7f49a2e289729ed03fe6c9dbd17ee"}
May 20 13:52:46.835054 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:46.834684    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0807f6a1-c649-418f-acde-1ea45476efe2","Type":"ContainerStarted","Data":"6957533dcb59613c8bedc212d2ab1646378e64530131ee3e590fb58323a262d8"}
May 20 13:52:47.363666 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:47.363565    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4cf4bd0-9845-4f94-bf86-154c26adcb33" path="/var/lib/kubelet/pods/c4cf4bd0-9845-4f94-bf86-154c26adcb33/volumes"
May 20 13:52:47.840464 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:47.840428    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0807f6a1-c649-418f-acde-1ea45476efe2","Type":"ContainerStarted","Data":"7a1ca9cf582eca85b57cccbad341056c9393fe98e0975cf757aba2e6838eb362"}
May 20 13:52:47.840464 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:47.840465    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0807f6a1-c649-418f-acde-1ea45476efe2","Type":"ContainerStarted","Data":"de9fb32dafd3ad899743f186173484d5d1c0784bd531dc34e2e497ad7a120c47"}
May 20 13:52:47.840888 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:47.840474    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0807f6a1-c649-418f-acde-1ea45476efe2","Type":"ContainerStarted","Data":"3306e328e3ee3c9e1716b21e5ac64664157d5dac820f3bd3879768138b2d4975"}
May 20 13:52:47.840888 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:47.840483    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0807f6a1-c649-418f-acde-1ea45476efe2","Type":"ContainerStarted","Data":"7e2a4a4326de28d1ef0446fc0b73fab1b77770017f600172dda09d3c3de9b14d"}
May 20 13:52:47.840888 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:47.840492    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0807f6a1-c649-418f-acde-1ea45476efe2","Type":"ContainerStarted","Data":"4177840d222c6941a093d0ce04838e2e4296c2f7719335b7db74ed3481c41ee6"}
May 20 13:52:47.840888 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:47.840500    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0807f6a1-c649-418f-acde-1ea45476efe2","Type":"ContainerStarted","Data":"62253e6a036b5e95f5aaf7b3f43b0d17c4b90e182a0a691caa7320f63a65efaa"}
May 20 13:52:48.582296 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.582246    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.582225478 podStartE2EDuration="3.582225478s" podCreationTimestamp="2026-05-20 13:52:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:52:47.886500686 +0000 UTC m=+135.124117332" watchObservedRunningTime="2026-05-20 13:52:48.582225478 +0000 UTC m=+135.819842117"
May 20 13:52:48.583508 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.583487    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6494fbb744-z97hh"]
May 20 13:52:48.586704 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.586689    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:48.590734 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:48.590700    2570 reflector.go:200] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"telemeter-client-serving-certs-ca-bundle\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" type="*v1.ConfigMap"
May 20 13:52:48.590866 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:48.590809    2570 reflector.go:200] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"telemeter-trusted-ca-bundle-8i12ta5c71j38\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" type="*v1.ConfigMap"
May 20 13:52:48.590866 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:48.590814    2570 reflector.go:200] "Failed to watch" err="failed to list *v1.Secret: secrets \"telemeter-client-dockercfg-br5qs\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-br5qs\"" type="*v1.Secret"
May 20 13:52:48.590866 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:48.590814    2570 reflector.go:200] "Failed to watch" err="failed to list *v1.Secret: secrets \"telemeter-client-tls\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" type="*v1.Secret"
May 20 13:52:48.590866 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.590826    2570 status_manager.go:895] "Failed to get status for pod" podUID="ad7632bd-8b00-44f6-af8b-66763c3175c7" pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh" err="pods \"telemeter-client-6494fbb744-z97hh\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object"
May 20 13:52:48.590866 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:48.590849    2570 reflector.go:200] "Failed to watch" err="failed to list *v1.Secret: secrets \"telemeter-client\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" type="*v1.Secret"
May 20 13:52:48.590866 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:48.590858    2570 reflector.go:200] "Failed to watch" err="failed to list *v1.Secret: secrets \"federate-client-certs\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" type="*v1.Secret"
May 20 13:52:48.591096 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:48.590927    2570 reflector.go:200] "Failed to watch" err="failed to list *v1.Secret: secrets \"telemeter-client-kube-rbac-proxy-config\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" type="*v1.Secret"
May 20 13:52:48.621082 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.621043    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6494fbb744-z97hh"]
May 20 13:52:48.706469 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.706422    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4qfh\" (UniqueName: \"kubernetes.io/projected/ad7632bd-8b00-44f6-af8b-66763c3175c7-kube-api-access-w4qfh\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:48.706669 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.706547    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ad7632bd-8b00-44f6-af8b-66763c3175c7-telemeter-client-tls\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:48.706669 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.706578    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad7632bd-8b00-44f6-af8b-66763c3175c7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:48.706809 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.706654    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7632bd-8b00-44f6-af8b-66763c3175c7-serving-certs-ca-bundle\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:48.706809 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.706717    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7632bd-8b00-44f6-af8b-66763c3175c7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:48.706809 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.706750    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ad7632bd-8b00-44f6-af8b-66763c3175c7-secret-telemeter-client\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:48.706809 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.706777    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ad7632bd-8b00-44f6-af8b-66763c3175c7-federate-client-tls\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:48.706981 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.706832    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad7632bd-8b00-44f6-af8b-66763c3175c7-metrics-client-ca\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:48.807510 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.807473    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ad7632bd-8b00-44f6-af8b-66763c3175c7-telemeter-client-tls\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:48.807510 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.807505    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad7632bd-8b00-44f6-af8b-66763c3175c7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:48.807785 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.807527    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7632bd-8b00-44f6-af8b-66763c3175c7-serving-certs-ca-bundle\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:48.807785 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.807553    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7632bd-8b00-44f6-af8b-66763c3175c7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:48.807785 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.807581    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ad7632bd-8b00-44f6-af8b-66763c3175c7-secret-telemeter-client\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:48.807785 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.807630    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ad7632bd-8b00-44f6-af8b-66763c3175c7-federate-client-tls\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:48.807785 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.807691    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad7632bd-8b00-44f6-af8b-66763c3175c7-metrics-client-ca\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:48.807785 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.807745    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w4qfh\" (UniqueName: \"kubernetes.io/projected/ad7632bd-8b00-44f6-af8b-66763c3175c7-kube-api-access-w4qfh\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:48.808321 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.808299    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad7632bd-8b00-44f6-af8b-66763c3175c7-metrics-client-ca\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:48.834153 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:48.834088    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4qfh\" (UniqueName: \"kubernetes.io/projected/ad7632bd-8b00-44f6-af8b-66763c3175c7-kube-api-access-w4qfh\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:49.543080 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:49.543043    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 20 13:52:49.547880 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:49.547854    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-br5qs\""
May 20 13:52:49.550260 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:49.550233    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad7632bd-8b00-44f6-af8b-66763c3175c7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:49.808678 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:49.808565    2570 secret.go:189] Couldn't get secret openshift-monitoring/federate-client-certs: failed to sync secret cache: timed out waiting for the condition
May 20 13:52:49.808678 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:49.808581    2570 configmap.go:193] Couldn't get configMap openshift-monitoring/telemeter-client-serving-certs-ca-bundle: failed to sync configmap cache: timed out waiting for the condition
May 20 13:52:49.808678 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:49.808565    2570 configmap.go:193] Couldn't get configMap openshift-monitoring/telemeter-trusted-ca-bundle-8i12ta5c71j38: failed to sync configmap cache: timed out waiting for the condition
May 20 13:52:49.808678 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:49.808671    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad7632bd-8b00-44f6-af8b-66763c3175c7-federate-client-tls podName:ad7632bd-8b00-44f6-af8b-66763c3175c7 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:50.308653945 +0000 UTC m=+137.546270570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "federate-client-tls" (UniqueName: "kubernetes.io/secret/ad7632bd-8b00-44f6-af8b-66763c3175c7-federate-client-tls") pod "telemeter-client-6494fbb744-z97hh" (UID: "ad7632bd-8b00-44f6-af8b-66763c3175c7") : failed to sync secret cache: timed out waiting for the condition
May 20 13:52:49.808678 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:49.808585    2570 secret.go:189] Couldn't get secret openshift-monitoring/telemeter-client: failed to sync secret cache: timed out waiting for the condition
May 20 13:52:49.808678 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:49.808686    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad7632bd-8b00-44f6-af8b-66763c3175c7-serving-certs-ca-bundle podName:ad7632bd-8b00-44f6-af8b-66763c3175c7 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:50.308680282 +0000 UTC m=+137.546296906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "serving-certs-ca-bundle" (UniqueName: "kubernetes.io/configmap/ad7632bd-8b00-44f6-af8b-66763c3175c7-serving-certs-ca-bundle") pod "telemeter-client-6494fbb744-z97hh" (UID: "ad7632bd-8b00-44f6-af8b-66763c3175c7") : failed to sync configmap cache: timed out waiting for the condition
May 20 13:52:49.809062 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:49.808587    2570 secret.go:189] Couldn't get secret openshift-monitoring/telemeter-client-tls: failed to sync secret cache: timed out waiting for the condition
May 20 13:52:49.809062 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:49.808701    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad7632bd-8b00-44f6-af8b-66763c3175c7-telemeter-trusted-ca-bundle podName:ad7632bd-8b00-44f6-af8b-66763c3175c7 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:50.308692862 +0000 UTC m=+137.546309492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "telemeter-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/ad7632bd-8b00-44f6-af8b-66763c3175c7-telemeter-trusted-ca-bundle") pod "telemeter-client-6494fbb744-z97hh" (UID: "ad7632bd-8b00-44f6-af8b-66763c3175c7") : failed to sync configmap cache: timed out waiting for the condition
May 20 13:52:49.809062 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:49.808716    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad7632bd-8b00-44f6-af8b-66763c3175c7-secret-telemeter-client podName:ad7632bd-8b00-44f6-af8b-66763c3175c7 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:50.308708747 +0000 UTC m=+137.546325392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-telemeter-client" (UniqueName: "kubernetes.io/secret/ad7632bd-8b00-44f6-af8b-66763c3175c7-secret-telemeter-client") pod "telemeter-client-6494fbb744-z97hh" (UID: "ad7632bd-8b00-44f6-af8b-66763c3175c7") : failed to sync secret cache: timed out waiting for the condition
May 20 13:52:49.809062 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:52:49.808727    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad7632bd-8b00-44f6-af8b-66763c3175c7-telemeter-client-tls podName:ad7632bd-8b00-44f6-af8b-66763c3175c7 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:50.308720902 +0000 UTC m=+137.546337528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "telemeter-client-tls" (UniqueName: "kubernetes.io/secret/ad7632bd-8b00-44f6-af8b-66763c3175c7-telemeter-client-tls") pod "telemeter-client-6494fbb744-z97hh" (UID: "ad7632bd-8b00-44f6-af8b-66763c3175c7") : failed to sync secret cache: timed out waiting for the condition
May 20 13:52:49.872821 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:49.872779    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 20 13:52:49.986904 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:49.986873    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 20 13:52:50.044968 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:50.044943    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 20 13:52:50.061725 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:50.061671    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 20 13:52:50.155277 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:50.155257    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 20 13:52:50.322097 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:50.322011    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ad7632bd-8b00-44f6-af8b-66763c3175c7-telemeter-client-tls\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:50.322097 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:50.322052    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7632bd-8b00-44f6-af8b-66763c3175c7-serving-certs-ca-bundle\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:50.322097 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:50.322097    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7632bd-8b00-44f6-af8b-66763c3175c7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:50.322313 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:50.322119    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ad7632bd-8b00-44f6-af8b-66763c3175c7-secret-telemeter-client\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:50.322313 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:50.322139    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ad7632bd-8b00-44f6-af8b-66763c3175c7-federate-client-tls\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:50.322941 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:50.322910    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7632bd-8b00-44f6-af8b-66763c3175c7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:50.323150 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:50.323129    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7632bd-8b00-44f6-af8b-66763c3175c7-serving-certs-ca-bundle\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:50.324502 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:50.324478    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ad7632bd-8b00-44f6-af8b-66763c3175c7-telemeter-client-tls\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:50.324635 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:50.324560    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ad7632bd-8b00-44f6-af8b-66763c3175c7-secret-telemeter-client\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:50.324635 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:50.324596    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ad7632bd-8b00-44f6-af8b-66763c3175c7-federate-client-tls\") pod \"telemeter-client-6494fbb744-z97hh\" (UID: \"ad7632bd-8b00-44f6-af8b-66763c3175c7\") " pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:50.396046 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:50.396009    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh"
May 20 13:52:50.561357 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:50.561320    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6494fbb744-z97hh"]
May 20 13:52:50.564054 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:52:50.564025    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7632bd_8b00_44f6_af8b_66763c3175c7.slice/crio-52cb192df772493762e2613ed61834792b1f1fa286b59f6f920f74eb30d9023b WatchSource:0}: Error finding container 52cb192df772493762e2613ed61834792b1f1fa286b59f6f920f74eb30d9023b: Status 404 returned error can't find the container with id 52cb192df772493762e2613ed61834792b1f1fa286b59f6f920f74eb30d9023b
May 20 13:52:50.852024 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:50.851986    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh" event={"ID":"ad7632bd-8b00-44f6-af8b-66763c3175c7","Type":"ContainerStarted","Data":"52cb192df772493762e2613ed61834792b1f1fa286b59f6f920f74eb30d9023b"}
May 20 13:52:52.859377 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:52.859343    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh" event={"ID":"ad7632bd-8b00-44f6-af8b-66763c3175c7","Type":"ContainerStarted","Data":"31ab13428f851ef66e00603bfc9817776137d51659e27670d3c9e41382abfa10"}
May 20 13:52:52.859377 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:52.859379    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh" event={"ID":"ad7632bd-8b00-44f6-af8b-66763c3175c7","Type":"ContainerStarted","Data":"ecfc8d98c44eb21c4805268861c52710cb5992ffdb7169de61baf396379b4b68"}
May 20 13:52:52.859852 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:52.859389    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh" event={"ID":"ad7632bd-8b00-44f6-af8b-66763c3175c7","Type":"ContainerStarted","Data":"93f0ae4128afb4ed3c010699d898d27e6c0805d9e6cae7b520c767a104d71255"}
May 20 13:52:52.889733 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:52.889673    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6494fbb744-z97hh" podStartSLOduration=3.076644408 podStartE2EDuration="4.8896573s" podCreationTimestamp="2026-05-20 13:52:48 +0000 UTC" firstStartedPulling="2026-05-20 13:52:50.565861721 +0000 UTC m=+137.803478345" lastFinishedPulling="2026-05-20 13:52:52.37887461 +0000 UTC m=+139.616491237" observedRunningTime="2026-05-20 13:52:52.886175985 +0000 UTC m=+140.123792654" watchObservedRunningTime="2026-05-20 13:52:52.8896573 +0000 UTC m=+140.127273945"
May 20 13:52:53.679779 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.679745    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-88c594f-6lg2t"]
May 20 13:52:53.682882 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.682864    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:53.743526 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.743487    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-88c594f-6lg2t"]
May 20 13:52:53.752779 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.752748    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-console-serving-cert\") pod \"console-88c594f-6lg2t\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") " pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:53.752946 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.752785    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-console-config\") pod \"console-88c594f-6lg2t\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") " pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:53.752946 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.752812    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-trusted-ca-bundle\") pod \"console-88c594f-6lg2t\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") " pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:53.752946 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.752874    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-oauth-serving-cert\") pod \"console-88c594f-6lg2t\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") " pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:53.752946 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.752890    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-console-oauth-config\") pod \"console-88c594f-6lg2t\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") " pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:53.752946 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.752943    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2gnm\" (UniqueName: \"kubernetes.io/projected/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-kube-api-access-x2gnm\") pod \"console-88c594f-6lg2t\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") " pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:53.753098 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.752960    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-service-ca\") pod \"console-88c594f-6lg2t\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") " pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:53.853804 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.853752    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x2gnm\" (UniqueName: \"kubernetes.io/projected/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-kube-api-access-x2gnm\") pod \"console-88c594f-6lg2t\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") " pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:53.853998 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.853815    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-service-ca\") pod \"console-88c594f-6lg2t\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") " pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:53.853998 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.853884    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-console-serving-cert\") pod \"console-88c594f-6lg2t\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") " pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:53.853998 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.853914    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-console-config\") pod \"console-88c594f-6lg2t\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") " pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:53.853998 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.853943    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-trusted-ca-bundle\") pod \"console-88c594f-6lg2t\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") " pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:53.854189 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.854001    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-oauth-serving-cert\") pod \"console-88c594f-6lg2t\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") " pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:53.854189 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.854027    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-console-oauth-config\") pod \"console-88c594f-6lg2t\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") " pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:53.854739 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.854705    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-service-ca\") pod \"console-88c594f-6lg2t\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") " pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:53.854739 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.854716    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-console-config\") pod \"console-88c594f-6lg2t\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") " pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:53.855100 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.855076    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-oauth-serving-cert\") pod \"console-88c594f-6lg2t\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") " pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:53.855390 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.855368    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-trusted-ca-bundle\") pod \"console-88c594f-6lg2t\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") " pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:53.856550 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.856527    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-console-serving-cert\") pod \"console-88c594f-6lg2t\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") " pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:53.856680 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.856661    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-console-oauth-config\") pod \"console-88c594f-6lg2t\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") " pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:53.864596 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.864575    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2gnm\" (UniqueName: \"kubernetes.io/projected/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-kube-api-access-x2gnm\") pod \"console-88c594f-6lg2t\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") " pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:53.992872 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:53.992769    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88c594f-6lg2t"
May 20 13:52:54.132109 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:54.132073    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-88c594f-6lg2t"]
May 20 13:52:54.135072 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:52:54.135035    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc0ceed_cd72_4c55_910a_8d362e6d0d86.slice/crio-d1f7c027932861401acb3170bfce4e0fe0bc8e36e119d161487f82d61b65297a WatchSource:0}: Error finding container d1f7c027932861401acb3170bfce4e0fe0bc8e36e119d161487f82d61b65297a: Status 404 returned error can't find the container with id d1f7c027932861401acb3170bfce4e0fe0bc8e36e119d161487f82d61b65297a
May 20 13:52:54.866310 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:54.866273    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88c594f-6lg2t" event={"ID":"fdc0ceed-cd72-4c55-910a-8d362e6d0d86","Type":"ContainerStarted","Data":"21c970f2ce6518080f997ee477e228f62b4f3f29e2e1656bb5f75ca49e3fb923"}
May 20 13:52:54.866310 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:54.866312    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88c594f-6lg2t" event={"ID":"fdc0ceed-cd72-4c55-910a-8d362e6d0d86","Type":"ContainerStarted","Data":"d1f7c027932861401acb3170bfce4e0fe0bc8e36e119d161487f82d61b65297a"}
May 20 13:52:54.892012 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:52:54.891958    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-88c594f-6lg2t" podStartSLOduration=1.891943009 podStartE2EDuration="1.891943009s" podCreationTimestamp="2026-05-20 13:52:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:52:54.890585226 +0000 UTC m=+142.128201882" watchObservedRunningTime="2026-05-20 13:52:54.891943009 +0000 UTC m=+142.129559655"
May 20 13:53:03.993420 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:03.993381    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-88c594f-6lg2t"
May 20 13:53:03.993420 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:03.993423    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-88c594f-6lg2t"
May 20 13:53:03.998286 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:03.998266    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-88c594f-6lg2t"
May 20 13:53:04.900071 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:04.899395    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-88c594f-6lg2t"
May 20 13:53:04.960837 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:04.960799    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8645b9c94b-9pftl"]
May 20 13:53:29.980844 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:29.980751    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-8645b9c94b-9pftl" podUID="ff0f998f-606c-4700-8737-218060a98a69" containerName="console" containerID="cri-o://a21b750e328839496abcf87f99700f102143398ed47e1e666373aa52858ccccc" gracePeriod=15
May 20 13:53:30.234379 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.234319    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8645b9c94b-9pftl_ff0f998f-606c-4700-8737-218060a98a69/console/0.log"
May 20 13:53:30.234379 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.234379    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:53:30.341528 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.341489    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-service-ca\") pod \"ff0f998f-606c-4700-8737-218060a98a69\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") "
May 20 13:53:30.341723 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.341545    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-trusted-ca-bundle\") pod \"ff0f998f-606c-4700-8737-218060a98a69\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") "
May 20 13:53:30.341723 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.341585    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff0f998f-606c-4700-8737-218060a98a69-console-oauth-config\") pod \"ff0f998f-606c-4700-8737-218060a98a69\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") "
May 20 13:53:30.341723 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.341677    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-console-config\") pod \"ff0f998f-606c-4700-8737-218060a98a69\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") "
May 20 13:53:30.341723 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.341714    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff0f998f-606c-4700-8737-218060a98a69-console-serving-cert\") pod \"ff0f998f-606c-4700-8737-218060a98a69\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") "
May 20 13:53:30.341920 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.341755    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-oauth-serving-cert\") pod \"ff0f998f-606c-4700-8737-218060a98a69\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") "
May 20 13:53:30.341920 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.341787    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrr4j\" (UniqueName: \"kubernetes.io/projected/ff0f998f-606c-4700-8737-218060a98a69-kube-api-access-qrr4j\") pod \"ff0f998f-606c-4700-8737-218060a98a69\" (UID: \"ff0f998f-606c-4700-8737-218060a98a69\") "
May 20 13:53:30.342018 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.341931    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ff0f998f-606c-4700-8737-218060a98a69" (UID: "ff0f998f-606c-4700-8737-218060a98a69"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:53:30.342075 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.342046    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-trusted-ca-bundle\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:53:30.342180 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.342143    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-console-config" (OuterVolumeSpecName: "console-config") pod "ff0f998f-606c-4700-8737-218060a98a69" (UID: "ff0f998f-606c-4700-8737-218060a98a69"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:53:30.342237 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.342187    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-service-ca" (OuterVolumeSpecName: "service-ca") pod "ff0f998f-606c-4700-8737-218060a98a69" (UID: "ff0f998f-606c-4700-8737-218060a98a69"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:53:30.342237 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.342158    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ff0f998f-606c-4700-8737-218060a98a69" (UID: "ff0f998f-606c-4700-8737-218060a98a69"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:53:30.343993 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.343966    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff0f998f-606c-4700-8737-218060a98a69-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ff0f998f-606c-4700-8737-218060a98a69" (UID: "ff0f998f-606c-4700-8737-218060a98a69"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:53:30.343993 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.343986    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff0f998f-606c-4700-8737-218060a98a69-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ff0f998f-606c-4700-8737-218060a98a69" (UID: "ff0f998f-606c-4700-8737-218060a98a69"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:53:30.344140 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.344024    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff0f998f-606c-4700-8737-218060a98a69-kube-api-access-qrr4j" (OuterVolumeSpecName: "kube-api-access-qrr4j") pod "ff0f998f-606c-4700-8737-218060a98a69" (UID: "ff0f998f-606c-4700-8737-218060a98a69"). InnerVolumeSpecName "kube-api-access-qrr4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:53:30.443028 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.442968    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff0f998f-606c-4700-8737-218060a98a69-console-oauth-config\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:53:30.443028 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.443021    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-console-config\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:53:30.443243 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.443055    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff0f998f-606c-4700-8737-218060a98a69-console-serving-cert\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:53:30.443243 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.443065    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-oauth-serving-cert\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:53:30.443243 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.443075    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrr4j\" (UniqueName: \"kubernetes.io/projected/ff0f998f-606c-4700-8737-218060a98a69-kube-api-access-qrr4j\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:53:30.443243 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.443085    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff0f998f-606c-4700-8737-218060a98a69-service-ca\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:53:30.967760 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.967732    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8645b9c94b-9pftl_ff0f998f-606c-4700-8737-218060a98a69/console/0.log"
May 20 13:53:30.967938 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.967770    2570 generic.go:358] "Generic (PLEG): container finished" podID="ff0f998f-606c-4700-8737-218060a98a69" containerID="a21b750e328839496abcf87f99700f102143398ed47e1e666373aa52858ccccc" exitCode=2
May 20 13:53:30.967938 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.967841    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8645b9c94b-9pftl"
May 20 13:53:30.967938 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.967863    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8645b9c94b-9pftl" event={"ID":"ff0f998f-606c-4700-8737-218060a98a69","Type":"ContainerDied","Data":"a21b750e328839496abcf87f99700f102143398ed47e1e666373aa52858ccccc"}
May 20 13:53:30.967938 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.967906    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8645b9c94b-9pftl" event={"ID":"ff0f998f-606c-4700-8737-218060a98a69","Type":"ContainerDied","Data":"9214a6425773c1a3795f3446f7bc0474ba2622054b694feeed6d5b7bd527ad82"}
May 20 13:53:30.967938 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.967935    2570 scope.go:117] "RemoveContainer" containerID="a21b750e328839496abcf87f99700f102143398ed47e1e666373aa52858ccccc"
May 20 13:53:30.976564 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.976547    2570 scope.go:117] "RemoveContainer" containerID="a21b750e328839496abcf87f99700f102143398ed47e1e666373aa52858ccccc"
May 20 13:53:30.976827 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:53:30.976809    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a21b750e328839496abcf87f99700f102143398ed47e1e666373aa52858ccccc\": container with ID starting with a21b750e328839496abcf87f99700f102143398ed47e1e666373aa52858ccccc not found: ID does not exist" containerID="a21b750e328839496abcf87f99700f102143398ed47e1e666373aa52858ccccc"
May 20 13:53:30.976894 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.976835    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21b750e328839496abcf87f99700f102143398ed47e1e666373aa52858ccccc"} err="failed to get container status \"a21b750e328839496abcf87f99700f102143398ed47e1e666373aa52858ccccc\": rpc error: code = NotFound desc = could not find container \"a21b750e328839496abcf87f99700f102143398ed47e1e666373aa52858ccccc\": container with ID starting with a21b750e328839496abcf87f99700f102143398ed47e1e666373aa52858ccccc not found: ID does not exist"
May 20 13:53:30.992874 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.992844    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8645b9c94b-9pftl"]
May 20 13:53:30.995088 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:30.995064    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8645b9c94b-9pftl"]
May 20 13:53:31.363161 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:53:31.363131    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff0f998f-606c-4700-8737-218060a98a69" path="/var/lib/kubelet/pods/ff0f998f-606c-4700-8737-218060a98a69/volumes"
May 20 13:54:07.517128 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.517096    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-665cbfb858-k9mf7"]
May 20 13:54:07.517595 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.517394    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff0f998f-606c-4700-8737-218060a98a69" containerName="console"
May 20 13:54:07.517595 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.517407    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff0f998f-606c-4700-8737-218060a98a69" containerName="console"
May 20 13:54:07.517595 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.517470    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="ff0f998f-606c-4700-8737-218060a98a69" containerName="console"
May 20 13:54:07.520212 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.520195    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.543174 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.543146    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-665cbfb858-k9mf7"]
May 20 13:54:07.636677 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.636632    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-trusted-ca-bundle\") pod \"console-665cbfb858-k9mf7\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") " pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.636848 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.636730    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-console-config\") pod \"console-665cbfb858-k9mf7\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") " pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.636848 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.636767    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d81baafa-32b8-4152-b4bd-f727b0f6a18a-console-serving-cert\") pod \"console-665cbfb858-k9mf7\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") " pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.636848 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.636791    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-service-ca\") pod \"console-665cbfb858-k9mf7\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") " pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.636848 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.636814    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fn5l\" (UniqueName: \"kubernetes.io/projected/d81baafa-32b8-4152-b4bd-f727b0f6a18a-kube-api-access-4fn5l\") pod \"console-665cbfb858-k9mf7\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") " pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.637046 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.636937    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d81baafa-32b8-4152-b4bd-f727b0f6a18a-console-oauth-config\") pod \"console-665cbfb858-k9mf7\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") " pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.637046 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.636971    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-oauth-serving-cert\") pod \"console-665cbfb858-k9mf7\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") " pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.737773 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.737736    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d81baafa-32b8-4152-b4bd-f727b0f6a18a-console-oauth-config\") pod \"console-665cbfb858-k9mf7\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") " pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.737773 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.737776    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-oauth-serving-cert\") pod \"console-665cbfb858-k9mf7\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") " pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.738022 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.737807    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-trusted-ca-bundle\") pod \"console-665cbfb858-k9mf7\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") " pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.738022 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.737850    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-console-config\") pod \"console-665cbfb858-k9mf7\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") " pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.738022 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.737869    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d81baafa-32b8-4152-b4bd-f727b0f6a18a-console-serving-cert\") pod \"console-665cbfb858-k9mf7\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") " pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.738022 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.737894    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-service-ca\") pod \"console-665cbfb858-k9mf7\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") " pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.738022 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.737926    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4fn5l\" (UniqueName: \"kubernetes.io/projected/d81baafa-32b8-4152-b4bd-f727b0f6a18a-kube-api-access-4fn5l\") pod \"console-665cbfb858-k9mf7\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") " pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.738648 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.738572    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-console-config\") pod \"console-665cbfb858-k9mf7\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") " pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.738648 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.738588    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-service-ca\") pod \"console-665cbfb858-k9mf7\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") " pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.738648 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.738575    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-oauth-serving-cert\") pod \"console-665cbfb858-k9mf7\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") " pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.738871 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.738836    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-trusted-ca-bundle\") pod \"console-665cbfb858-k9mf7\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") " pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.740886 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.740865    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d81baafa-32b8-4152-b4bd-f727b0f6a18a-console-oauth-config\") pod \"console-665cbfb858-k9mf7\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") " pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.740979 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.740937    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d81baafa-32b8-4152-b4bd-f727b0f6a18a-console-serving-cert\") pod \"console-665cbfb858-k9mf7\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") " pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.750308 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.750285    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fn5l\" (UniqueName: \"kubernetes.io/projected/d81baafa-32b8-4152-b4bd-f727b0f6a18a-kube-api-access-4fn5l\") pod \"console-665cbfb858-k9mf7\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") " pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.828942 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.828903    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:07.969325 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:07.969292    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-665cbfb858-k9mf7"]
May 20 13:54:07.972098 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:54:07.972071    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81baafa_32b8_4152_b4bd_f727b0f6a18a.slice/crio-7bf9005d8d92695240f85d91876cf56248f89ae48775a8a1f6e94c86dccbba0e WatchSource:0}: Error finding container 7bf9005d8d92695240f85d91876cf56248f89ae48775a8a1f6e94c86dccbba0e: Status 404 returned error can't find the container with id 7bf9005d8d92695240f85d91876cf56248f89ae48775a8a1f6e94c86dccbba0e
May 20 13:54:08.072276 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:08.072243    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-665cbfb858-k9mf7" event={"ID":"d81baafa-32b8-4152-b4bd-f727b0f6a18a","Type":"ContainerStarted","Data":"d6c48ffbf24292631089d2b505608ed6fb84bb5e29eb8f7d860801afdb147957"}
May 20 13:54:08.072276 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:08.072281    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-665cbfb858-k9mf7" event={"ID":"d81baafa-32b8-4152-b4bd-f727b0f6a18a","Type":"ContainerStarted","Data":"7bf9005d8d92695240f85d91876cf56248f89ae48775a8a1f6e94c86dccbba0e"}
May 20 13:54:08.102443 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:08.102330    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-665cbfb858-k9mf7" podStartSLOduration=1.102310933 podStartE2EDuration="1.102310933s" podCreationTimestamp="2026-05-20 13:54:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:54:08.101082224 +0000 UTC m=+215.338698871" watchObservedRunningTime="2026-05-20 13:54:08.102310933 +0000 UTC m=+215.339927581"
May 20 13:54:17.829839 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:17.829798    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:17.829839 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:17.829847    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:17.834417 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:17.834393    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:18.105793 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:18.105715    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:54:18.183059 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:18.183024    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-88c594f-6lg2t"]
May 20 13:54:25.785754 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:25.785714    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-kvcpv"]
May 20 13:54:25.788777 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:25.788760    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kvcpv"
May 20 13:54:25.791316 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:25.791297    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 20 13:54:25.813994 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:25.813962    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kvcpv"]
May 20 13:54:25.872992 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:25.872958    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a7a2cce0-9de9-4b66-83f3-2c58f1656ed1-dbus\") pod \"global-pull-secret-syncer-kvcpv\" (UID: \"a7a2cce0-9de9-4b66-83f3-2c58f1656ed1\") " pod="kube-system/global-pull-secret-syncer-kvcpv"
May 20 13:54:25.872992 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:25.872995    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a7a2cce0-9de9-4b66-83f3-2c58f1656ed1-kubelet-config\") pod \"global-pull-secret-syncer-kvcpv\" (UID: \"a7a2cce0-9de9-4b66-83f3-2c58f1656ed1\") " pod="kube-system/global-pull-secret-syncer-kvcpv"
May 20 13:54:25.873194 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:25.873039    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7a2cce0-9de9-4b66-83f3-2c58f1656ed1-original-pull-secret\") pod \"global-pull-secret-syncer-kvcpv\" (UID: \"a7a2cce0-9de9-4b66-83f3-2c58f1656ed1\") " pod="kube-system/global-pull-secret-syncer-kvcpv"
May 20 13:54:25.973820 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:25.973780    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7a2cce0-9de9-4b66-83f3-2c58f1656ed1-original-pull-secret\") pod \"global-pull-secret-syncer-kvcpv\" (UID: \"a7a2cce0-9de9-4b66-83f3-2c58f1656ed1\") " pod="kube-system/global-pull-secret-syncer-kvcpv"
May 20 13:54:25.974020 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:25.973868    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a7a2cce0-9de9-4b66-83f3-2c58f1656ed1-dbus\") pod \"global-pull-secret-syncer-kvcpv\" (UID: \"a7a2cce0-9de9-4b66-83f3-2c58f1656ed1\") " pod="kube-system/global-pull-secret-syncer-kvcpv"
May 20 13:54:25.974020 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:25.973901    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a7a2cce0-9de9-4b66-83f3-2c58f1656ed1-kubelet-config\") pod \"global-pull-secret-syncer-kvcpv\" (UID: \"a7a2cce0-9de9-4b66-83f3-2c58f1656ed1\") " pod="kube-system/global-pull-secret-syncer-kvcpv"
May 20 13:54:25.974020 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:25.974003    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a7a2cce0-9de9-4b66-83f3-2c58f1656ed1-kubelet-config\") pod \"global-pull-secret-syncer-kvcpv\" (UID: \"a7a2cce0-9de9-4b66-83f3-2c58f1656ed1\") " pod="kube-system/global-pull-secret-syncer-kvcpv"
May 20 13:54:25.974143 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:25.974058    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a7a2cce0-9de9-4b66-83f3-2c58f1656ed1-dbus\") pod \"global-pull-secret-syncer-kvcpv\" (UID: \"a7a2cce0-9de9-4b66-83f3-2c58f1656ed1\") " pod="kube-system/global-pull-secret-syncer-kvcpv"
May 20 13:54:25.976207 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:25.976190    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7a2cce0-9de9-4b66-83f3-2c58f1656ed1-original-pull-secret\") pod \"global-pull-secret-syncer-kvcpv\" (UID: \"a7a2cce0-9de9-4b66-83f3-2c58f1656ed1\") " pod="kube-system/global-pull-secret-syncer-kvcpv"
May 20 13:54:26.098812 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:26.098702    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kvcpv"
May 20 13:54:26.242094 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:26.242063    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kvcpv"]
May 20 13:54:26.245777 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:54:26.245746    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a2cce0_9de9_4b66_83f3_2c58f1656ed1.slice/crio-36625d659270f74624686f8fb40f4efc84041f34cae7b8f1428f779754a0b972 WatchSource:0}: Error finding container 36625d659270f74624686f8fb40f4efc84041f34cae7b8f1428f779754a0b972: Status 404 returned error can't find the container with id 36625d659270f74624686f8fb40f4efc84041f34cae7b8f1428f779754a0b972
May 20 13:54:27.127525 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:27.127487    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kvcpv" event={"ID":"a7a2cce0-9de9-4b66-83f3-2c58f1656ed1","Type":"ContainerStarted","Data":"36625d659270f74624686f8fb40f4efc84041f34cae7b8f1428f779754a0b972"}
May 20 13:54:30.144987 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:30.144945    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kvcpv" event={"ID":"a7a2cce0-9de9-4b66-83f3-2c58f1656ed1","Type":"ContainerStarted","Data":"711a8352149a6a9f0948c1c463ef68ad81585fc537cdd05dc6373d5f91a38025"}
May 20 13:54:30.174477 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:30.174379    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kvcpv" podStartSLOduration=1.498688676 podStartE2EDuration="5.174364181s" podCreationTimestamp="2026-05-20 13:54:25 +0000 UTC" firstStartedPulling="2026-05-20 13:54:26.247290696 +0000 UTC m=+233.484907321" lastFinishedPulling="2026-05-20 13:54:29.922966197 +0000 UTC m=+237.160582826" observedRunningTime="2026-05-20 13:54:30.173808538 +0000 UTC m=+237.411425195" watchObservedRunningTime="2026-05-20 13:54:30.174364181 +0000 UTC m=+237.411980891"
May 20 13:54:43.202203 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.202131    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-88c594f-6lg2t" podUID="fdc0ceed-cd72-4c55-910a-8d362e6d0d86" containerName="console" containerID="cri-o://21c970f2ce6518080f997ee477e228f62b4f3f29e2e1656bb5f75ca49e3fb923" gracePeriod=15
May 20 13:54:43.442140 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.442114    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-88c594f-6lg2t_fdc0ceed-cd72-4c55-910a-8d362e6d0d86/console/0.log"
May 20 13:54:43.442265 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.442174    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88c594f-6lg2t"
May 20 13:54:43.522054 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.522023    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-console-serving-cert\") pod \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") "
May 20 13:54:43.522221 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.522065    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-service-ca\") pod \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") "
May 20 13:54:43.522221 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.522103    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-oauth-serving-cert\") pod \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") "
May 20 13:54:43.522221 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.522164    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-console-oauth-config\") pod \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") "
May 20 13:54:43.522221 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.522202    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2gnm\" (UniqueName: \"kubernetes.io/projected/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-kube-api-access-x2gnm\") pod \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") "
May 20 13:54:43.522433 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.522278    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-trusted-ca-bundle\") pod \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") "
May 20 13:54:43.522433 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.522309    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-console-config\") pod \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\" (UID: \"fdc0ceed-cd72-4c55-910a-8d362e6d0d86\") "
May 20 13:54:43.522665 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.522595    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fdc0ceed-cd72-4c55-910a-8d362e6d0d86" (UID: "fdc0ceed-cd72-4c55-910a-8d362e6d0d86"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:54:43.522665 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.522648    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-service-ca" (OuterVolumeSpecName: "service-ca") pod "fdc0ceed-cd72-4c55-910a-8d362e6d0d86" (UID: "fdc0ceed-cd72-4c55-910a-8d362e6d0d86"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:54:43.522903 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.522772    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fdc0ceed-cd72-4c55-910a-8d362e6d0d86" (UID: "fdc0ceed-cd72-4c55-910a-8d362e6d0d86"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:54:43.522903 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.522824    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-console-config" (OuterVolumeSpecName: "console-config") pod "fdc0ceed-cd72-4c55-910a-8d362e6d0d86" (UID: "fdc0ceed-cd72-4c55-910a-8d362e6d0d86"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:54:43.524658 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.524587    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fdc0ceed-cd72-4c55-910a-8d362e6d0d86" (UID: "fdc0ceed-cd72-4c55-910a-8d362e6d0d86"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:43.524658 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.524642    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-kube-api-access-x2gnm" (OuterVolumeSpecName: "kube-api-access-x2gnm") pod "fdc0ceed-cd72-4c55-910a-8d362e6d0d86" (UID: "fdc0ceed-cd72-4c55-910a-8d362e6d0d86"). InnerVolumeSpecName "kube-api-access-x2gnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:54:43.524785 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.524683    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fdc0ceed-cd72-4c55-910a-8d362e6d0d86" (UID: "fdc0ceed-cd72-4c55-910a-8d362e6d0d86"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:43.623075 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.623019    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-console-config\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:54:43.623075 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.623070    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-console-serving-cert\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:54:43.623075 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.623080    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-service-ca\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:54:43.623075 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.623090    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-oauth-serving-cert\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:54:43.623344 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.623099    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-console-oauth-config\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:54:43.623344 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.623108    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2gnm\" (UniqueName: \"kubernetes.io/projected/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-kube-api-access-x2gnm\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:54:43.623344 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:43.623117    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdc0ceed-cd72-4c55-910a-8d362e6d0d86-trusted-ca-bundle\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:54:44.183684 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:44.183657    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-88c594f-6lg2t_fdc0ceed-cd72-4c55-910a-8d362e6d0d86/console/0.log"
May 20 13:54:44.183850 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:44.183699    2570 generic.go:358] "Generic (PLEG): container finished" podID="fdc0ceed-cd72-4c55-910a-8d362e6d0d86" containerID="21c970f2ce6518080f997ee477e228f62b4f3f29e2e1656bb5f75ca49e3fb923" exitCode=2
May 20 13:54:44.183850 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:44.183733    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88c594f-6lg2t" event={"ID":"fdc0ceed-cd72-4c55-910a-8d362e6d0d86","Type":"ContainerDied","Data":"21c970f2ce6518080f997ee477e228f62b4f3f29e2e1656bb5f75ca49e3fb923"}
May 20 13:54:44.183850 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:44.183781    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88c594f-6lg2t" event={"ID":"fdc0ceed-cd72-4c55-910a-8d362e6d0d86","Type":"ContainerDied","Data":"d1f7c027932861401acb3170bfce4e0fe0bc8e36e119d161487f82d61b65297a"}
May 20 13:54:44.183850 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:44.183784    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88c594f-6lg2t"
May 20 13:54:44.183985 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:44.183848    2570 scope.go:117] "RemoveContainer" containerID="21c970f2ce6518080f997ee477e228f62b4f3f29e2e1656bb5f75ca49e3fb923"
May 20 13:54:44.191832 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:44.191798    2570 scope.go:117] "RemoveContainer" containerID="21c970f2ce6518080f997ee477e228f62b4f3f29e2e1656bb5f75ca49e3fb923"
May 20 13:54:44.192062 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:54:44.192045    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c970f2ce6518080f997ee477e228f62b4f3f29e2e1656bb5f75ca49e3fb923\": container with ID starting with 21c970f2ce6518080f997ee477e228f62b4f3f29e2e1656bb5f75ca49e3fb923 not found: ID does not exist" containerID="21c970f2ce6518080f997ee477e228f62b4f3f29e2e1656bb5f75ca49e3fb923"
May 20 13:54:44.192121 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:44.192069    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c970f2ce6518080f997ee477e228f62b4f3f29e2e1656bb5f75ca49e3fb923"} err="failed to get container status \"21c970f2ce6518080f997ee477e228f62b4f3f29e2e1656bb5f75ca49e3fb923\": rpc error: code = NotFound desc = could not find container \"21c970f2ce6518080f997ee477e228f62b4f3f29e2e1656bb5f75ca49e3fb923\": container with ID starting with 21c970f2ce6518080f997ee477e228f62b4f3f29e2e1656bb5f75ca49e3fb923 not found: ID does not exist"
May 20 13:54:44.210472 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:44.210445    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-88c594f-6lg2t"]
May 20 13:54:44.216724 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:44.216702    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-88c594f-6lg2t"]
May 20 13:54:45.363020 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:45.362991    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdc0ceed-cd72-4c55-910a-8d362e6d0d86" path="/var/lib/kubelet/pods/fdc0ceed-cd72-4c55-910a-8d362e6d0d86/volumes"
May 20 13:54:49.030388 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:49.030351    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2"]
May 20 13:54:49.030869 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:49.030796    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdc0ceed-cd72-4c55-910a-8d362e6d0d86" containerName="console"
May 20 13:54:49.030869 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:49.030815    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdc0ceed-cd72-4c55-910a-8d362e6d0d86" containerName="console"
May 20 13:54:49.030986 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:49.030900    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="fdc0ceed-cd72-4c55-910a-8d362e6d0d86" containerName="console"
May 20 13:54:49.032814 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:49.032794    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2"
May 20 13:54:49.036941 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:49.036919    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 20 13:54:49.037050 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:49.036919    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vvzwl\""
May 20 13:54:49.037955 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:49.037935    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 20 13:54:49.054634 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:49.054591    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2"]
May 20 13:54:49.173869 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:49.173839    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6faf21f1-74e1-4046-b855-3ed45221c01c-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2\" (UID: \"6faf21f1-74e1-4046-b855-3ed45221c01c\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2"
May 20 13:54:49.174102 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:49.173891    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6faf21f1-74e1-4046-b855-3ed45221c01c-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2\" (UID: \"6faf21f1-74e1-4046-b855-3ed45221c01c\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2"
May 20 13:54:49.174102 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:49.173916    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctp2m\" (UniqueName: \"kubernetes.io/projected/6faf21f1-74e1-4046-b855-3ed45221c01c-kube-api-access-ctp2m\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2\" (UID: \"6faf21f1-74e1-4046-b855-3ed45221c01c\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2"
May 20 13:54:49.274935 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:49.274878    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6faf21f1-74e1-4046-b855-3ed45221c01c-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2\" (UID: \"6faf21f1-74e1-4046-b855-3ed45221c01c\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2"
May 20 13:54:49.274935 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:49.274943    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6faf21f1-74e1-4046-b855-3ed45221c01c-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2\" (UID: \"6faf21f1-74e1-4046-b855-3ed45221c01c\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2"
May 20 13:54:49.275153 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:49.274968    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ctp2m\" (UniqueName: \"kubernetes.io/projected/6faf21f1-74e1-4046-b855-3ed45221c01c-kube-api-access-ctp2m\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2\" (UID: \"6faf21f1-74e1-4046-b855-3ed45221c01c\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2"
May 20 13:54:49.275302 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:49.275283    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6faf21f1-74e1-4046-b855-3ed45221c01c-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2\" (UID: \"6faf21f1-74e1-4046-b855-3ed45221c01c\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2"
May 20 13:54:49.275350 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:49.275331    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6faf21f1-74e1-4046-b855-3ed45221c01c-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2\" (UID: \"6faf21f1-74e1-4046-b855-3ed45221c01c\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2"
May 20 13:54:49.287152 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:49.287080    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctp2m\" (UniqueName: \"kubernetes.io/projected/6faf21f1-74e1-4046-b855-3ed45221c01c-kube-api-access-ctp2m\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2\" (UID: \"6faf21f1-74e1-4046-b855-3ed45221c01c\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2"
May 20 13:54:49.341876 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:49.341831    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2"
May 20 13:54:49.505367 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:49.505284    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2"]
May 20 13:54:49.509947 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:54:49.509919    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6faf21f1_74e1_4046_b855_3ed45221c01c.slice/crio-c687b9320e828c8aaf398251fe8e5f747ca45c3df63c38b9bd4434576ba09759 WatchSource:0}: Error finding container c687b9320e828c8aaf398251fe8e5f747ca45c3df63c38b9bd4434576ba09759: Status 404 returned error can't find the container with id c687b9320e828c8aaf398251fe8e5f747ca45c3df63c38b9bd4434576ba09759
May 20 13:54:50.203022 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:50.202989    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2" event={"ID":"6faf21f1-74e1-4046-b855-3ed45221c01c","Type":"ContainerStarted","Data":"c687b9320e828c8aaf398251fe8e5f747ca45c3df63c38b9bd4434576ba09759"}
May 20 13:54:55.218997 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:55.218959    2570 generic.go:358] "Generic (PLEG): container finished" podID="6faf21f1-74e1-4046-b855-3ed45221c01c" containerID="487885f05b2a3c7648899602a9694793a8ca92073f6b132d91a74fef331bbae2" exitCode=0
May 20 13:54:55.219429 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:54:55.219048    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2" event={"ID":"6faf21f1-74e1-4046-b855-3ed45221c01c","Type":"ContainerDied","Data":"487885f05b2a3c7648899602a9694793a8ca92073f6b132d91a74fef331bbae2"}
May 20 13:55:09.263801 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:09.263761    2570 generic.go:358] "Generic (PLEG): container finished" podID="6faf21f1-74e1-4046-b855-3ed45221c01c" containerID="0878e2ca4b97a46412098a013b3344e1d547840ec7a7c6b4983a5209ae8ab6f7" exitCode=0
May 20 13:55:09.264273 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:09.263822    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2" event={"ID":"6faf21f1-74e1-4046-b855-3ed45221c01c","Type":"ContainerDied","Data":"0878e2ca4b97a46412098a013b3344e1d547840ec7a7c6b4983a5209ae8ab6f7"}
May 20 13:55:16.285154 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:16.285118    2570 generic.go:358] "Generic (PLEG): container finished" podID="6faf21f1-74e1-4046-b855-3ed45221c01c" containerID="6fd9ccaf2716bcc74406078d45a5350bfd713c15f44eb3ae9ecadcfde6778e65" exitCode=0
May 20 13:55:16.285664 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:16.285165    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2" event={"ID":"6faf21f1-74e1-4046-b855-3ed45221c01c","Type":"ContainerDied","Data":"6fd9ccaf2716bcc74406078d45a5350bfd713c15f44eb3ae9ecadcfde6778e65"}
May 20 13:55:17.409891 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:17.409864    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2"
May 20 13:55:17.505852 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:17.505810    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6faf21f1-74e1-4046-b855-3ed45221c01c-util\") pod \"6faf21f1-74e1-4046-b855-3ed45221c01c\" (UID: \"6faf21f1-74e1-4046-b855-3ed45221c01c\") "
May 20 13:55:17.506020 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:17.505865    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctp2m\" (UniqueName: \"kubernetes.io/projected/6faf21f1-74e1-4046-b855-3ed45221c01c-kube-api-access-ctp2m\") pod \"6faf21f1-74e1-4046-b855-3ed45221c01c\" (UID: \"6faf21f1-74e1-4046-b855-3ed45221c01c\") "
May 20 13:55:17.506020 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:17.505937    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6faf21f1-74e1-4046-b855-3ed45221c01c-bundle\") pod \"6faf21f1-74e1-4046-b855-3ed45221c01c\" (UID: \"6faf21f1-74e1-4046-b855-3ed45221c01c\") "
May 20 13:55:17.506510 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:17.506482    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6faf21f1-74e1-4046-b855-3ed45221c01c-bundle" (OuterVolumeSpecName: "bundle") pod "6faf21f1-74e1-4046-b855-3ed45221c01c" (UID: "6faf21f1-74e1-4046-b855-3ed45221c01c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 13:55:17.508224 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:17.508200    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6faf21f1-74e1-4046-b855-3ed45221c01c-kube-api-access-ctp2m" (OuterVolumeSpecName: "kube-api-access-ctp2m") pod "6faf21f1-74e1-4046-b855-3ed45221c01c" (UID: "6faf21f1-74e1-4046-b855-3ed45221c01c"). InnerVolumeSpecName "kube-api-access-ctp2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:55:17.512106 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:17.512082    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6faf21f1-74e1-4046-b855-3ed45221c01c-util" (OuterVolumeSpecName: "util") pod "6faf21f1-74e1-4046-b855-3ed45221c01c" (UID: "6faf21f1-74e1-4046-b855-3ed45221c01c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 13:55:17.607370 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:17.607270    2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6faf21f1-74e1-4046-b855-3ed45221c01c-util\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:55:17.607370 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:17.607315    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctp2m\" (UniqueName: \"kubernetes.io/projected/6faf21f1-74e1-4046-b855-3ed45221c01c-kube-api-access-ctp2m\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:55:17.607370 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:17.607326    2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6faf21f1-74e1-4046-b855-3ed45221c01c-bundle\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:55:18.292366 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:18.292337    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2" event={"ID":"6faf21f1-74e1-4046-b855-3ed45221c01c","Type":"ContainerDied","Data":"c687b9320e828c8aaf398251fe8e5f747ca45c3df63c38b9bd4434576ba09759"}
May 20 13:55:18.292366 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:18.292367    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c687b9320e828c8aaf398251fe8e5f747ca45c3df63c38b9bd4434576ba09759"
May 20 13:55:18.292568 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:18.292389    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4r5n2"
May 20 13:55:25.478095 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:25.478062    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vtllj"]
May 20 13:55:25.478552 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:25.478428    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6faf21f1-74e1-4046-b855-3ed45221c01c" containerName="util"
May 20 13:55:25.478552 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:25.478445    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6faf21f1-74e1-4046-b855-3ed45221c01c" containerName="util"
May 20 13:55:25.478552 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:25.478460    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6faf21f1-74e1-4046-b855-3ed45221c01c" containerName="extract"
May 20 13:55:25.478552 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:25.478469    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6faf21f1-74e1-4046-b855-3ed45221c01c" containerName="extract"
May 20 13:55:25.478552 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:25.478479    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6faf21f1-74e1-4046-b855-3ed45221c01c" containerName="pull"
May 20 13:55:25.478552 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:25.478488    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6faf21f1-74e1-4046-b855-3ed45221c01c" containerName="pull"
May 20 13:55:25.478552 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:25.478552    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="6faf21f1-74e1-4046-b855-3ed45221c01c" containerName="extract"
May 20 13:55:25.480533 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:25.480514    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vtllj"
May 20 13:55:25.493793 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:25.493774    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
May 20 13:55:25.494552 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:25.494528    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-2zvkn\""
May 20 13:55:25.494691 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:25.494580    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
May 20 13:55:25.494848 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:25.494832    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
May 20 13:55:25.569387 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:25.569348    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xnvx\" (UniqueName: \"kubernetes.io/projected/04e4ae59-3283-4f83-adb1-4393dda2c83a-kube-api-access-2xnvx\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-vtllj\" (UID: \"04e4ae59-3283-4f83-adb1-4393dda2c83a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vtllj"
May 20 13:55:25.569534 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:25.569417    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/04e4ae59-3283-4f83-adb1-4393dda2c83a-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-vtllj\" (UID: \"04e4ae59-3283-4f83-adb1-4393dda2c83a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vtllj"
May 20 13:55:25.577627 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:25.577576    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vtllj"]
May 20 13:55:25.670646 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:25.670598    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/04e4ae59-3283-4f83-adb1-4393dda2c83a-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-vtllj\" (UID: \"04e4ae59-3283-4f83-adb1-4393dda2c83a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vtllj"
May 20 13:55:25.670824 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:25.670674    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2xnvx\" (UniqueName: \"kubernetes.io/projected/04e4ae59-3283-4f83-adb1-4393dda2c83a-kube-api-access-2xnvx\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-vtllj\" (UID: \"04e4ae59-3283-4f83-adb1-4393dda2c83a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vtllj"
May 20 13:55:25.673064 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:25.673035    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/04e4ae59-3283-4f83-adb1-4393dda2c83a-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-vtllj\" (UID: \"04e4ae59-3283-4f83-adb1-4393dda2c83a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vtllj"
May 20 13:55:25.697533 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:25.697504    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xnvx\" (UniqueName: \"kubernetes.io/projected/04e4ae59-3283-4f83-adb1-4393dda2c83a-kube-api-access-2xnvx\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-vtllj\" (UID: \"04e4ae59-3283-4f83-adb1-4393dda2c83a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vtllj"
May 20 13:55:25.791410 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:25.791370    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vtllj"
May 20 13:55:26.092533 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:26.092450    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vtllj"]
May 20 13:55:26.156516 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:55:26.156488    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e4ae59_3283_4f83_adb1_4393dda2c83a.slice/crio-255774e223d46e1f73b6e7f6648bb8ae685b49b43a3b1dfd429eed78b50030e2 WatchSource:0}: Error finding container 255774e223d46e1f73b6e7f6648bb8ae685b49b43a3b1dfd429eed78b50030e2: Status 404 returned error can't find the container with id 255774e223d46e1f73b6e7f6648bb8ae685b49b43a3b1dfd429eed78b50030e2
May 20 13:55:26.316181 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:26.316147    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vtllj" event={"ID":"04e4ae59-3283-4f83-adb1-4393dda2c83a","Type":"ContainerStarted","Data":"255774e223d46e1f73b6e7f6648bb8ae685b49b43a3b1dfd429eed78b50030e2"}
May 20 13:55:30.332366 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:30.332327    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vtllj" event={"ID":"04e4ae59-3283-4f83-adb1-4393dda2c83a","Type":"ContainerStarted","Data":"e3597ed082015de38b7bfc156793815afe87fac6e12b5fd6070d545dd0d8371e"}
May 20 13:55:30.332782 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:30.332644    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vtllj"
May 20 13:55:30.392517 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:30.392451    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vtllj" podStartSLOduration=1.9753711950000001 podStartE2EDuration="5.392431969s" podCreationTimestamp="2026-05-20 13:55:25 +0000 UTC" firstStartedPulling="2026-05-20 13:55:26.158105267 +0000 UTC m=+293.395721891" lastFinishedPulling="2026-05-20 13:55:29.575166025 +0000 UTC m=+296.812782665" observedRunningTime="2026-05-20 13:55:30.389794438 +0000 UTC m=+297.627411086" watchObservedRunningTime="2026-05-20 13:55:30.392431969 +0000 UTC m=+297.630048619"
May 20 13:55:30.783780 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:30.783743    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-fsmtc"]
May 20 13:55:30.786307 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:30.786276    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-fsmtc"
May 20 13:55:30.793320 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:30.793297    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
May 20 13:55:30.794081 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:30.794063    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-h6b9s\""
May 20 13:55:30.799023 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:30.799010    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
May 20 13:55:30.816582 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:30.816543    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmhgk\" (UniqueName: \"kubernetes.io/projected/92121bce-e934-4ccc-9bd6-449dd303d646-kube-api-access-wmhgk\") pod \"keda-operator-ffbb595cb-fsmtc\" (UID: \"92121bce-e934-4ccc-9bd6-449dd303d646\") " pod="openshift-keda/keda-operator-ffbb595cb-fsmtc"
May 20 13:55:30.816729 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:30.816631    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/92121bce-e934-4ccc-9bd6-449dd303d646-cabundle0\") pod \"keda-operator-ffbb595cb-fsmtc\" (UID: \"92121bce-e934-4ccc-9bd6-449dd303d646\") " pod="openshift-keda/keda-operator-ffbb595cb-fsmtc"
May 20 13:55:30.816773 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:30.816739    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/92121bce-e934-4ccc-9bd6-449dd303d646-certificates\") pod \"keda-operator-ffbb595cb-fsmtc\" (UID: \"92121bce-e934-4ccc-9bd6-449dd303d646\") " pod="openshift-keda/keda-operator-ffbb595cb-fsmtc"
May 20 13:55:30.853347 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:30.853311    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-fsmtc"]
May 20 13:55:30.917302 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:30.917259    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/92121bce-e934-4ccc-9bd6-449dd303d646-cabundle0\") pod \"keda-operator-ffbb595cb-fsmtc\" (UID: \"92121bce-e934-4ccc-9bd6-449dd303d646\") " pod="openshift-keda/keda-operator-ffbb595cb-fsmtc"
May 20 13:55:30.917505 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:30.917358    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/92121bce-e934-4ccc-9bd6-449dd303d646-certificates\") pod \"keda-operator-ffbb595cb-fsmtc\" (UID: \"92121bce-e934-4ccc-9bd6-449dd303d646\") " pod="openshift-keda/keda-operator-ffbb595cb-fsmtc"
May 20 13:55:30.917505 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:30.917397    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wmhgk\" (UniqueName: \"kubernetes.io/projected/92121bce-e934-4ccc-9bd6-449dd303d646-kube-api-access-wmhgk\") pod \"keda-operator-ffbb595cb-fsmtc\" (UID: \"92121bce-e934-4ccc-9bd6-449dd303d646\") " pod="openshift-keda/keda-operator-ffbb595cb-fsmtc"
May 20 13:55:30.917651 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:30.917508    2570 projected.go:264] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
May 20 13:55:30.917651 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:30.917532    2570 secret.go:281] references non-existent secret key: ca.crt
May 20 13:55:30.917651 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:30.917544    2570 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 20 13:55:30.917651 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:30.917561    2570 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-fsmtc: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
May 20 13:55:30.917853 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:30.917680    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92121bce-e934-4ccc-9bd6-449dd303d646-certificates podName:92121bce-e934-4ccc-9bd6-449dd303d646 nodeName:}" failed. No retries permitted until 2026-05-20 13:55:31.417655122 +0000 UTC m=+298.655271761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/92121bce-e934-4ccc-9bd6-449dd303d646-certificates") pod "keda-operator-ffbb595cb-fsmtc" (UID: "92121bce-e934-4ccc-9bd6-449dd303d646") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
May 20 13:55:30.918161 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:30.918112    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/92121bce-e934-4ccc-9bd6-449dd303d646-cabundle0\") pod \"keda-operator-ffbb595cb-fsmtc\" (UID: \"92121bce-e934-4ccc-9bd6-449dd303d646\") " pod="openshift-keda/keda-operator-ffbb595cb-fsmtc"
May 20 13:55:31.000932 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.000893    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmhgk\" (UniqueName: \"kubernetes.io/projected/92121bce-e934-4ccc-9bd6-449dd303d646-kube-api-access-wmhgk\") pod \"keda-operator-ffbb595cb-fsmtc\" (UID: \"92121bce-e934-4ccc-9bd6-449dd303d646\") " pod="openshift-keda/keda-operator-ffbb595cb-fsmtc"
May 20 13:55:31.294719 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.294682    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc"]
May 20 13:55:31.296815 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.296799    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc"
May 20 13:55:31.299873 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.299853    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
May 20 13:55:31.318775 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.318742    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc"]
May 20 13:55:31.423365 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.423328    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9rp9\" (UniqueName: \"kubernetes.io/projected/cf493e82-a5fd-43fb-b019-5f31486d6336-kube-api-access-v9rp9\") pod \"keda-metrics-apiserver-7c9f485588-whvdc\" (UID: \"cf493e82-a5fd-43fb-b019-5f31486d6336\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc"
May 20 13:55:31.423860 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.423403    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/92121bce-e934-4ccc-9bd6-449dd303d646-certificates\") pod \"keda-operator-ffbb595cb-fsmtc\" (UID: \"92121bce-e934-4ccc-9bd6-449dd303d646\") " pod="openshift-keda/keda-operator-ffbb595cb-fsmtc"
May 20 13:55:31.423860 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.423430    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cf493e82-a5fd-43fb-b019-5f31486d6336-certificates\") pod \"keda-metrics-apiserver-7c9f485588-whvdc\" (UID: \"cf493e82-a5fd-43fb-b019-5f31486d6336\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc"
May 20 13:55:31.423860 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.423453    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/cf493e82-a5fd-43fb-b019-5f31486d6336-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-whvdc\" (UID: \"cf493e82-a5fd-43fb-b019-5f31486d6336\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc"
May 20 13:55:31.423860 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:31.423519    2570 secret.go:281] references non-existent secret key: ca.crt
May 20 13:55:31.423860 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:31.423544    2570 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 20 13:55:31.423860 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:31.423557    2570 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-fsmtc: references non-existent secret key: ca.crt
May 20 13:55:31.423860 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:31.423605    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92121bce-e934-4ccc-9bd6-449dd303d646-certificates podName:92121bce-e934-4ccc-9bd6-449dd303d646 nodeName:}" failed. No retries permitted until 2026-05-20 13:55:32.423590913 +0000 UTC m=+299.661207542 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/92121bce-e934-4ccc-9bd6-449dd303d646-certificates") pod "keda-operator-ffbb595cb-fsmtc" (UID: "92121bce-e934-4ccc-9bd6-449dd303d646") : references non-existent secret key: ca.crt
May 20 13:55:31.524534 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.524498    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v9rp9\" (UniqueName: \"kubernetes.io/projected/cf493e82-a5fd-43fb-b019-5f31486d6336-kube-api-access-v9rp9\") pod \"keda-metrics-apiserver-7c9f485588-whvdc\" (UID: \"cf493e82-a5fd-43fb-b019-5f31486d6336\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc"
May 20 13:55:31.524708 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.524565    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cf493e82-a5fd-43fb-b019-5f31486d6336-certificates\") pod \"keda-metrics-apiserver-7c9f485588-whvdc\" (UID: \"cf493e82-a5fd-43fb-b019-5f31486d6336\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc"
May 20 13:55:31.524708 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.524589    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/cf493e82-a5fd-43fb-b019-5f31486d6336-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-whvdc\" (UID: \"cf493e82-a5fd-43fb-b019-5f31486d6336\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc"
May 20 13:55:31.524780 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:31.524705    2570 secret.go:281] references non-existent secret key: tls.crt
May 20 13:55:31.524780 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:31.524724    2570 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 20 13:55:31.524780 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:31.524747    2570 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc: references non-existent secret key: tls.crt
May 20 13:55:31.524871 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:31.524811    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf493e82-a5fd-43fb-b019-5f31486d6336-certificates podName:cf493e82-a5fd-43fb-b019-5f31486d6336 nodeName:}" failed. No retries permitted until 2026-05-20 13:55:32.02478918 +0000 UTC m=+299.262405804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/cf493e82-a5fd-43fb-b019-5f31486d6336-certificates") pod "keda-metrics-apiserver-7c9f485588-whvdc" (UID: "cf493e82-a5fd-43fb-b019-5f31486d6336") : references non-existent secret key: tls.crt
May 20 13:55:31.525002 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.524983    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/cf493e82-a5fd-43fb-b019-5f31486d6336-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-whvdc\" (UID: \"cf493e82-a5fd-43fb-b019-5f31486d6336\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc"
May 20 13:55:31.538900 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.538877    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9rp9\" (UniqueName: \"kubernetes.io/projected/cf493e82-a5fd-43fb-b019-5f31486d6336-kube-api-access-v9rp9\") pod \"keda-metrics-apiserver-7c9f485588-whvdc\" (UID: \"cf493e82-a5fd-43fb-b019-5f31486d6336\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc"
May 20 13:55:31.730430 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.730335    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-cf49989db-jpg27"]
May 20 13:55:31.732372 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.732353    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-jpg27"
May 20 13:55:31.756622 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.756581    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
May 20 13:55:31.807588 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.807546    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-jpg27"]
May 20 13:55:31.827674 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.827638    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7wtd\" (UniqueName: \"kubernetes.io/projected/c00c989e-bac0-41a9-a0a3-ac19283019fe-kube-api-access-s7wtd\") pod \"keda-admission-cf49989db-jpg27\" (UID: \"c00c989e-bac0-41a9-a0a3-ac19283019fe\") " pod="openshift-keda/keda-admission-cf49989db-jpg27"
May 20 13:55:31.828024 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.828001    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c00c989e-bac0-41a9-a0a3-ac19283019fe-certificates\") pod \"keda-admission-cf49989db-jpg27\" (UID: \"c00c989e-bac0-41a9-a0a3-ac19283019fe\") " pod="openshift-keda/keda-admission-cf49989db-jpg27"
May 20 13:55:31.928943 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.928903    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wtd\" (UniqueName: \"kubernetes.io/projected/c00c989e-bac0-41a9-a0a3-ac19283019fe-kube-api-access-s7wtd\") pod \"keda-admission-cf49989db-jpg27\" (UID: \"c00c989e-bac0-41a9-a0a3-ac19283019fe\") " pod="openshift-keda/keda-admission-cf49989db-jpg27"
May 20 13:55:31.929234 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.929206    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c00c989e-bac0-41a9-a0a3-ac19283019fe-certificates\") pod \"keda-admission-cf49989db-jpg27\" (UID: \"c00c989e-bac0-41a9-a0a3-ac19283019fe\") " pod="openshift-keda/keda-admission-cf49989db-jpg27"
May 20 13:55:31.931931 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.931909    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c00c989e-bac0-41a9-a0a3-ac19283019fe-certificates\") pod \"keda-admission-cf49989db-jpg27\" (UID: \"c00c989e-bac0-41a9-a0a3-ac19283019fe\") " pod="openshift-keda/keda-admission-cf49989db-jpg27"
May 20 13:55:31.956356 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:31.956331    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7wtd\" (UniqueName: \"kubernetes.io/projected/c00c989e-bac0-41a9-a0a3-ac19283019fe-kube-api-access-s7wtd\") pod \"keda-admission-cf49989db-jpg27\" (UID: \"c00c989e-bac0-41a9-a0a3-ac19283019fe\") " pod="openshift-keda/keda-admission-cf49989db-jpg27"
May 20 13:55:32.030671 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:32.030633    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cf493e82-a5fd-43fb-b019-5f31486d6336-certificates\") pod \"keda-metrics-apiserver-7c9f485588-whvdc\" (UID: \"cf493e82-a5fd-43fb-b019-5f31486d6336\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc"
May 20 13:55:32.030847 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:32.030722    2570 secret.go:281] references non-existent secret key: tls.crt
May 20 13:55:32.030847 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:32.030744    2570 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 20 13:55:32.030847 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:32.030766    2570 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc: references non-existent secret key: tls.crt
May 20 13:55:32.030847 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:32.030820    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf493e82-a5fd-43fb-b019-5f31486d6336-certificates podName:cf493e82-a5fd-43fb-b019-5f31486d6336 nodeName:}" failed. No retries permitted until 2026-05-20 13:55:33.030805902 +0000 UTC m=+300.268422528 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/cf493e82-a5fd-43fb-b019-5f31486d6336-certificates") pod "keda-metrics-apiserver-7c9f485588-whvdc" (UID: "cf493e82-a5fd-43fb-b019-5f31486d6336") : references non-existent secret key: tls.crt
May 20 13:55:32.042582 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:32.042545    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-jpg27"
May 20 13:55:32.209048 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:55:32.209000    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00c989e_bac0_41a9_a0a3_ac19283019fe.slice/crio-f7269d9a51d3381d476a60d723e015dcf166370ffbbdbd1b0090f729166d3b29 WatchSource:0}: Error finding container f7269d9a51d3381d476a60d723e015dcf166370ffbbdbd1b0090f729166d3b29: Status 404 returned error can't find the container with id f7269d9a51d3381d476a60d723e015dcf166370ffbbdbd1b0090f729166d3b29
May 20 13:55:32.216701 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:32.216673    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-jpg27"]
May 20 13:55:32.339700 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:32.339576    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-jpg27" event={"ID":"c00c989e-bac0-41a9-a0a3-ac19283019fe","Type":"ContainerStarted","Data":"f7269d9a51d3381d476a60d723e015dcf166370ffbbdbd1b0090f729166d3b29"}
May 20 13:55:32.434526 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:32.434488    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/92121bce-e934-4ccc-9bd6-449dd303d646-certificates\") pod \"keda-operator-ffbb595cb-fsmtc\" (UID: \"92121bce-e934-4ccc-9bd6-449dd303d646\") " pod="openshift-keda/keda-operator-ffbb595cb-fsmtc"
May 20 13:55:32.435058 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:32.434664    2570 secret.go:281] references non-existent secret key: ca.crt
May 20 13:55:32.435058 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:32.434684    2570 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 20 13:55:32.435058 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:32.434694    2570 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-fsmtc: references non-existent secret key: ca.crt
May 20 13:55:32.435058 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:32.434760    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92121bce-e934-4ccc-9bd6-449dd303d646-certificates podName:92121bce-e934-4ccc-9bd6-449dd303d646 nodeName:}" failed. No retries permitted until 2026-05-20 13:55:34.434745521 +0000 UTC m=+301.672362151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/92121bce-e934-4ccc-9bd6-449dd303d646-certificates") pod "keda-operator-ffbb595cb-fsmtc" (UID: "92121bce-e934-4ccc-9bd6-449dd303d646") : references non-existent secret key: ca.crt
May 20 13:55:33.043908 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:33.039918    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cf493e82-a5fd-43fb-b019-5f31486d6336-certificates\") pod \"keda-metrics-apiserver-7c9f485588-whvdc\" (UID: \"cf493e82-a5fd-43fb-b019-5f31486d6336\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc"
May 20 13:55:33.043908 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:33.040170    2570 secret.go:281] references non-existent secret key: tls.crt
May 20 13:55:33.043908 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:33.040188    2570 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 20 13:55:33.043908 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:33.040220    2570 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc: references non-existent secret key: tls.crt
May 20 13:55:33.043908 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:33.040296    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf493e82-a5fd-43fb-b019-5f31486d6336-certificates podName:cf493e82-a5fd-43fb-b019-5f31486d6336 nodeName:}" failed. No retries permitted until 2026-05-20 13:55:35.040270209 +0000 UTC m=+302.277886847 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/cf493e82-a5fd-43fb-b019-5f31486d6336-certificates") pod "keda-metrics-apiserver-7c9f485588-whvdc" (UID: "cf493e82-a5fd-43fb-b019-5f31486d6336") : references non-existent secret key: tls.crt
May 20 13:55:33.253604 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:33.253575    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 13:55:33.253830 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:33.253810    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 13:55:33.261507 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:33.261485    2570 kubelet.go:1628] "Image garbage collection succeeded"
May 20 13:55:34.347747 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:34.347711    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-jpg27" event={"ID":"c00c989e-bac0-41a9-a0a3-ac19283019fe","Type":"ContainerStarted","Data":"764f851f17a06349aef91fd726735ff5514a1af73d3b2de6c0c3990a2dfc9ba1"}
May 20 13:55:34.348218 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:34.347866    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-cf49989db-jpg27"
May 20 13:55:34.380227 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:34.380168    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-admission-cf49989db-jpg27" podStartSLOduration=1.639840765 podStartE2EDuration="3.380147944s" podCreationTimestamp="2026-05-20 13:55:31 +0000 UTC" firstStartedPulling="2026-05-20 13:55:32.210583501 +0000 UTC m=+299.448200126" lastFinishedPulling="2026-05-20 13:55:33.95089068 +0000 UTC m=+301.188507305" observedRunningTime="2026-05-20 13:55:34.377507405 +0000 UTC m=+301.615124053" watchObservedRunningTime="2026-05-20 13:55:34.380147944 +0000 UTC m=+301.617764592"
May 20 13:55:34.452013 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:34.451967    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/92121bce-e934-4ccc-9bd6-449dd303d646-certificates\") pod \"keda-operator-ffbb595cb-fsmtc\" (UID: \"92121bce-e934-4ccc-9bd6-449dd303d646\") " pod="openshift-keda/keda-operator-ffbb595cb-fsmtc"
May 20 13:55:34.452013 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:34.452001    2570 secret.go:281] references non-existent secret key: ca.crt
May 20 13:55:34.452013 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:34.452018    2570 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 20 13:55:34.452311 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:34.452030    2570 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-fsmtc: references non-existent secret key: ca.crt
May 20 13:55:34.452311 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:34.452086    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92121bce-e934-4ccc-9bd6-449dd303d646-certificates podName:92121bce-e934-4ccc-9bd6-449dd303d646 nodeName:}" failed. No retries permitted until 2026-05-20 13:55:38.45206737 +0000 UTC m=+305.689684009 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/92121bce-e934-4ccc-9bd6-449dd303d646-certificates") pod "keda-operator-ffbb595cb-fsmtc" (UID: "92121bce-e934-4ccc-9bd6-449dd303d646") : references non-existent secret key: ca.crt
May 20 13:55:35.056056 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:35.056010    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cf493e82-a5fd-43fb-b019-5f31486d6336-certificates\") pod \"keda-metrics-apiserver-7c9f485588-whvdc\" (UID: \"cf493e82-a5fd-43fb-b019-5f31486d6336\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc"
May 20 13:55:35.056242 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:35.056164    2570 secret.go:281] references non-existent secret key: tls.crt
May 20 13:55:35.056242 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:35.056186    2570 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 20 13:55:35.056242 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:35.056208    2570 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc: references non-existent secret key: tls.crt
May 20 13:55:35.056355 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:55:35.056265    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf493e82-a5fd-43fb-b019-5f31486d6336-certificates podName:cf493e82-a5fd-43fb-b019-5f31486d6336 nodeName:}" failed. No retries permitted until 2026-05-20 13:55:39.056251342 +0000 UTC m=+306.293867970 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/cf493e82-a5fd-43fb-b019-5f31486d6336-certificates") pod "keda-metrics-apiserver-7c9f485588-whvdc" (UID: "cf493e82-a5fd-43fb-b019-5f31486d6336") : references non-existent secret key: tls.crt
May 20 13:55:38.485537 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:38.485493    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/92121bce-e934-4ccc-9bd6-449dd303d646-certificates\") pod \"keda-operator-ffbb595cb-fsmtc\" (UID: \"92121bce-e934-4ccc-9bd6-449dd303d646\") " pod="openshift-keda/keda-operator-ffbb595cb-fsmtc"
May 20 13:55:38.488071 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:38.488049    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/92121bce-e934-4ccc-9bd6-449dd303d646-certificates\") pod \"keda-operator-ffbb595cb-fsmtc\" (UID: \"92121bce-e934-4ccc-9bd6-449dd303d646\") " pod="openshift-keda/keda-operator-ffbb595cb-fsmtc"
May 20 13:55:38.596407 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:38.596370    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-fsmtc"
May 20 13:55:38.738087 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:38.738058    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-fsmtc"]
May 20 13:55:38.740357 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:55:38.740325    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92121bce_e934_4ccc_9bd6_449dd303d646.slice/crio-adef0b95dec192ea35056e905ffec929dbe2e342e11398313f8b7e71d176190e WatchSource:0}: Error finding container adef0b95dec192ea35056e905ffec929dbe2e342e11398313f8b7e71d176190e: Status 404 returned error can't find the container with id adef0b95dec192ea35056e905ffec929dbe2e342e11398313f8b7e71d176190e
May 20 13:55:38.741690 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:38.741672    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 13:55:39.092210 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:39.092173    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cf493e82-a5fd-43fb-b019-5f31486d6336-certificates\") pod \"keda-metrics-apiserver-7c9f485588-whvdc\" (UID: \"cf493e82-a5fd-43fb-b019-5f31486d6336\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc"
May 20 13:55:39.094841 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:39.094818    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cf493e82-a5fd-43fb-b019-5f31486d6336-certificates\") pod \"keda-metrics-apiserver-7c9f485588-whvdc\" (UID: \"cf493e82-a5fd-43fb-b019-5f31486d6336\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc"
May 20 13:55:39.107672 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:39.107639    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc"
May 20 13:55:39.251438 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:39.251391    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc"]
May 20 13:55:39.254773 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:55:39.254746    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf493e82_a5fd_43fb_b019_5f31486d6336.slice/crio-d0d1a8367865f96b07a46d918eb674907e1786ded7326162838d730cdd497c9c WatchSource:0}: Error finding container d0d1a8367865f96b07a46d918eb674907e1786ded7326162838d730cdd497c9c: Status 404 returned error can't find the container with id d0d1a8367865f96b07a46d918eb674907e1786ded7326162838d730cdd497c9c
May 20 13:55:39.363841 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:39.363742    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc" event={"ID":"cf493e82-a5fd-43fb-b019-5f31486d6336","Type":"ContainerStarted","Data":"d0d1a8367865f96b07a46d918eb674907e1786ded7326162838d730cdd497c9c"}
May 20 13:55:39.363991 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:39.363943    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-fsmtc" event={"ID":"92121bce-e934-4ccc-9bd6-449dd303d646","Type":"ContainerStarted","Data":"adef0b95dec192ea35056e905ffec929dbe2e342e11398313f8b7e71d176190e"}
May 20 13:55:43.377254 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:43.377156    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc" event={"ID":"cf493e82-a5fd-43fb-b019-5f31486d6336","Type":"ContainerStarted","Data":"2f3fc99a84a1a7f8a47a1b4bf89eebff65b0dd564a1c1b0e10ece2ee960c80c6"}
May 20 13:55:43.377254 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:43.377220    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc"
May 20 13:55:43.378639 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:43.378599    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-fsmtc" event={"ID":"92121bce-e934-4ccc-9bd6-449dd303d646","Type":"ContainerStarted","Data":"ee52710c5e625fed8a5a81882c00d39c7a692086bb6ed66d1d2525bc723bf695"}
May 20 13:55:43.378808 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:43.378791    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-ffbb595cb-fsmtc"
May 20 13:55:43.459534 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:43.459481    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-operator-ffbb595cb-fsmtc" podStartSLOduration=9.089884617 podStartE2EDuration="13.459464488s" podCreationTimestamp="2026-05-20 13:55:30 +0000 UTC" firstStartedPulling="2026-05-20 13:55:38.741796928 +0000 UTC m=+305.979413552" lastFinishedPulling="2026-05-20 13:55:43.111376798 +0000 UTC m=+310.348993423" observedRunningTime="2026-05-20 13:55:43.449862694 +0000 UTC m=+310.687479408" watchObservedRunningTime="2026-05-20 13:55:43.459464488 +0000 UTC m=+310.697081117"
May 20 13:55:51.338555 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:51.338524    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vtllj"
May 20 13:55:51.369031 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:51.368984    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc" podStartSLOduration=16.52178191 podStartE2EDuration="20.368969492s" podCreationTimestamp="2026-05-20 13:55:31 +0000 UTC" firstStartedPulling="2026-05-20 13:55:39.256202034 +0000 UTC m=+306.493818659" lastFinishedPulling="2026-05-20 13:55:43.103389613 +0000 UTC m=+310.341006241" observedRunningTime="2026-05-20 13:55:43.487286861 +0000 UTC m=+310.724903509" watchObservedRunningTime="2026-05-20 13:55:51.368969492 +0000 UTC m=+318.606586139"
May 20 13:55:54.387733 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:54.387693    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-whvdc"
May 20 13:55:55.353602 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:55:55.353565    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-cf49989db-jpg27"
May 20 13:56:04.384707 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:04.384669    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-ffbb595cb-fsmtc"
May 20 13:56:44.981682 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:44.981585    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-867cffc775-hv55f"]
May 20 13:56:44.984158 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:44.984132    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867cffc775-hv55f"
May 20 13:56:44.985208 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:44.985184    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-68cc5db7c4-gh9gs"]
May 20 13:56:44.987306 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:44.987290    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-68cc5db7c4-gh9gs"
May 20 13:56:44.987787 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:44.987771    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
May 20 13:56:44.987891 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:44.987873    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 20 13:56:44.987960 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:44.987877    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-ndnrr\""
May 20 13:56:44.987960 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:44.987899    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 20 13:56:44.990980 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:44.990963    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
May 20 13:56:44.991132 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:44.991115    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-gkfd7\""
May 20 13:56:45.001032 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.000986    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-867cffc775-hv55f"]
May 20 13:56:45.002326 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.002303    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-68cc5db7c4-gh9gs"]
May 20 13:56:45.013401 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.013372    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-79f457656b-dt4x8"]
May 20 13:56:45.015949 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.015931    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-dt4x8"
May 20 13:56:45.018393 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.018373    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 20 13:56:45.018702 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.018685    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dln44\""
May 20 13:56:45.025930 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.025910    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-dt4x8"]
May 20 13:56:45.137369 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.137333    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2df218e5-e163-4608-9c2a-b80f6f2b82a9-data\") pod \"seaweedfs-79f457656b-dt4x8\" (UID: \"2df218e5-e163-4608-9c2a-b80f6f2b82a9\") " pod="kserve/seaweedfs-79f457656b-dt4x8"
May 20 13:56:45.137570 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.137398    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxd5b\" (UniqueName: \"kubernetes.io/projected/2df218e5-e163-4608-9c2a-b80f6f2b82a9-kube-api-access-fxd5b\") pod \"seaweedfs-79f457656b-dt4x8\" (UID: \"2df218e5-e163-4608-9c2a-b80f6f2b82a9\") " pod="kserve/seaweedfs-79f457656b-dt4x8"
May 20 13:56:45.137570 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.137468    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2503b4b5-50c4-40d2-8e74-6de98c022740-cert\") pod \"llmisvc-controller-manager-68cc5db7c4-gh9gs\" (UID: \"2503b4b5-50c4-40d2-8e74-6de98c022740\") " pod="kserve/llmisvc-controller-manager-68cc5db7c4-gh9gs"
May 20 13:56:45.137570 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.137501    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb493e78-f437-47cb-b403-20e336a77c91-cert\") pod \"kserve-controller-manager-867cffc775-hv55f\" (UID: \"fb493e78-f437-47cb-b403-20e336a77c91\") " pod="kserve/kserve-controller-manager-867cffc775-hv55f"
May 20 13:56:45.137570 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.137526    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsslt\" (UniqueName: \"kubernetes.io/projected/fb493e78-f437-47cb-b403-20e336a77c91-kube-api-access-wsslt\") pod \"kserve-controller-manager-867cffc775-hv55f\" (UID: \"fb493e78-f437-47cb-b403-20e336a77c91\") " pod="kserve/kserve-controller-manager-867cffc775-hv55f"
May 20 13:56:45.137570 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.137552    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7j5f\" (UniqueName: \"kubernetes.io/projected/2503b4b5-50c4-40d2-8e74-6de98c022740-kube-api-access-b7j5f\") pod \"llmisvc-controller-manager-68cc5db7c4-gh9gs\" (UID: \"2503b4b5-50c4-40d2-8e74-6de98c022740\") " pod="kserve/llmisvc-controller-manager-68cc5db7c4-gh9gs"
May 20 13:56:45.238361 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.238264    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fxd5b\" (UniqueName: \"kubernetes.io/projected/2df218e5-e163-4608-9c2a-b80f6f2b82a9-kube-api-access-fxd5b\") pod \"seaweedfs-79f457656b-dt4x8\" (UID: \"2df218e5-e163-4608-9c2a-b80f6f2b82a9\") " pod="kserve/seaweedfs-79f457656b-dt4x8"
May 20 13:56:45.238361 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.238314    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2503b4b5-50c4-40d2-8e74-6de98c022740-cert\") pod \"llmisvc-controller-manager-68cc5db7c4-gh9gs\" (UID: \"2503b4b5-50c4-40d2-8e74-6de98c022740\") " pod="kserve/llmisvc-controller-manager-68cc5db7c4-gh9gs"
May 20 13:56:45.238559 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.238432    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb493e78-f437-47cb-b403-20e336a77c91-cert\") pod \"kserve-controller-manager-867cffc775-hv55f\" (UID: \"fb493e78-f437-47cb-b403-20e336a77c91\") " pod="kserve/kserve-controller-manager-867cffc775-hv55f"
May 20 13:56:45.238559 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.238477    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wsslt\" (UniqueName: \"kubernetes.io/projected/fb493e78-f437-47cb-b403-20e336a77c91-kube-api-access-wsslt\") pod \"kserve-controller-manager-867cffc775-hv55f\" (UID: \"fb493e78-f437-47cb-b403-20e336a77c91\") " pod="kserve/kserve-controller-manager-867cffc775-hv55f"
May 20 13:56:45.238559 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.238511    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b7j5f\" (UniqueName: \"kubernetes.io/projected/2503b4b5-50c4-40d2-8e74-6de98c022740-kube-api-access-b7j5f\") pod \"llmisvc-controller-manager-68cc5db7c4-gh9gs\" (UID: \"2503b4b5-50c4-40d2-8e74-6de98c022740\") " pod="kserve/llmisvc-controller-manager-68cc5db7c4-gh9gs"
May 20 13:56:45.238719 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.238570    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2df218e5-e163-4608-9c2a-b80f6f2b82a9-data\") pod \"seaweedfs-79f457656b-dt4x8\" (UID: \"2df218e5-e163-4608-9c2a-b80f6f2b82a9\") " pod="kserve/seaweedfs-79f457656b-dt4x8"
May 20 13:56:45.239007 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.238981    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2df218e5-e163-4608-9c2a-b80f6f2b82a9-data\") pod \"seaweedfs-79f457656b-dt4x8\" (UID: \"2df218e5-e163-4608-9c2a-b80f6f2b82a9\") " pod="kserve/seaweedfs-79f457656b-dt4x8"
May 20 13:56:45.240970 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.240939    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2503b4b5-50c4-40d2-8e74-6de98c022740-cert\") pod \"llmisvc-controller-manager-68cc5db7c4-gh9gs\" (UID: \"2503b4b5-50c4-40d2-8e74-6de98c022740\") " pod="kserve/llmisvc-controller-manager-68cc5db7c4-gh9gs"
May 20 13:56:45.241058 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.240978    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb493e78-f437-47cb-b403-20e336a77c91-cert\") pod \"kserve-controller-manager-867cffc775-hv55f\" (UID: \"fb493e78-f437-47cb-b403-20e336a77c91\") " pod="kserve/kserve-controller-manager-867cffc775-hv55f"
May 20 13:56:45.251917 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.251890    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7j5f\" (UniqueName: \"kubernetes.io/projected/2503b4b5-50c4-40d2-8e74-6de98c022740-kube-api-access-b7j5f\") pod \"llmisvc-controller-manager-68cc5db7c4-gh9gs\" (UID: \"2503b4b5-50c4-40d2-8e74-6de98c022740\") " pod="kserve/llmisvc-controller-manager-68cc5db7c4-gh9gs"
May 20 13:56:45.252186 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.252163    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxd5b\" (UniqueName: \"kubernetes.io/projected/2df218e5-e163-4608-9c2a-b80f6f2b82a9-kube-api-access-fxd5b\") pod \"seaweedfs-79f457656b-dt4x8\" (UID: \"2df218e5-e163-4608-9c2a-b80f6f2b82a9\") " pod="kserve/seaweedfs-79f457656b-dt4x8"
May 20 13:56:45.253911 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.253894    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsslt\" (UniqueName: \"kubernetes.io/projected/fb493e78-f437-47cb-b403-20e336a77c91-kube-api-access-wsslt\") pod \"kserve-controller-manager-867cffc775-hv55f\" (UID: \"fb493e78-f437-47cb-b403-20e336a77c91\") " pod="kserve/kserve-controller-manager-867cffc775-hv55f"
May 20 13:56:45.296922 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.296888    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867cffc775-hv55f"
May 20 13:56:45.303748 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.303722    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-68cc5db7c4-gh9gs"
May 20 13:56:45.325446 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.325415    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-dt4x8"
May 20 13:56:45.487485 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.487453    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-68cc5db7c4-gh9gs"]
May 20 13:56:45.490646 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:56:45.490558    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2503b4b5_50c4_40d2_8e74_6de98c022740.slice/crio-a49123e2ee879f76e07fd1b6b648f6451754743434f86a784a4d4e98405bd11e WatchSource:0}: Error finding container a49123e2ee879f76e07fd1b6b648f6451754743434f86a784a4d4e98405bd11e: Status 404 returned error can't find the container with id a49123e2ee879f76e07fd1b6b648f6451754743434f86a784a4d4e98405bd11e
May 20 13:56:45.574901 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.574861    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-68cc5db7c4-gh9gs" event={"ID":"2503b4b5-50c4-40d2-8e74-6de98c022740","Type":"ContainerStarted","Data":"a49123e2ee879f76e07fd1b6b648f6451754743434f86a784a4d4e98405bd11e"}
May 20 13:56:45.708288 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.708250    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-867cffc775-hv55f"]
May 20 13:56:45.710400 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:45.710365    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-dt4x8"]
May 20 13:56:45.711583 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:56:45.711558    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb493e78_f437_47cb_b403_20e336a77c91.slice/crio-659bf4311de984c5a4bb30f2f18690e0dbc887e2238dd83c20fe3f44ff8556af WatchSource:0}: Error finding container 659bf4311de984c5a4bb30f2f18690e0dbc887e2238dd83c20fe3f44ff8556af: Status 404 returned error can't find the container with id 659bf4311de984c5a4bb30f2f18690e0dbc887e2238dd83c20fe3f44ff8556af
May 20 13:56:45.713417 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:56:45.713394    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df218e5_e163_4608_9c2a_b80f6f2b82a9.slice/crio-778f3daad4a18ca00db6060dc53356e78a174d2faa6292fce56747fda27745da WatchSource:0}: Error finding container 778f3daad4a18ca00db6060dc53356e78a174d2faa6292fce56747fda27745da: Status 404 returned error can't find the container with id 778f3daad4a18ca00db6060dc53356e78a174d2faa6292fce56747fda27745da
May 20 13:56:46.586010 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:46.585965    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-dt4x8" event={"ID":"2df218e5-e163-4608-9c2a-b80f6f2b82a9","Type":"ContainerStarted","Data":"778f3daad4a18ca00db6060dc53356e78a174d2faa6292fce56747fda27745da"}
May 20 13:56:46.587192 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:46.587163    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867cffc775-hv55f" event={"ID":"fb493e78-f437-47cb-b403-20e336a77c91","Type":"ContainerStarted","Data":"659bf4311de984c5a4bb30f2f18690e0dbc887e2238dd83c20fe3f44ff8556af"}
May 20 13:56:49.600678 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:49.600638    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867cffc775-hv55f" event={"ID":"fb493e78-f437-47cb-b403-20e336a77c91","Type":"ContainerStarted","Data":"7666ecfe4508214fc3bbe193105cd2257b151ab5d02942ee5ecd7fd6757d3567"}
May 20 13:56:49.601123 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:49.600859    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-867cffc775-hv55f"
May 20 13:56:49.630503 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:49.630435    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-867cffc775-hv55f" podStartSLOduration=1.998319373 podStartE2EDuration="5.630416726s" podCreationTimestamp="2026-05-20 13:56:44 +0000 UTC" firstStartedPulling="2026-05-20 13:56:45.71332371 +0000 UTC m=+372.950940336" lastFinishedPulling="2026-05-20 13:56:49.345421064 +0000 UTC m=+376.583037689" observedRunningTime="2026-05-20 13:56:49.628391392 +0000 UTC m=+376.866008053" watchObservedRunningTime="2026-05-20 13:56:49.630416726 +0000 UTC m=+376.868033376"
May 20 13:56:50.606685 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:50.606648    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-dt4x8" event={"ID":"2df218e5-e163-4608-9c2a-b80f6f2b82a9","Type":"ContainerStarted","Data":"05c7ab1f25877e4a6470ace95df6b4385a031d618c69c5ef5ec547135d62502c"}
May 20 13:56:50.607149 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:50.606862    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-79f457656b-dt4x8"
May 20 13:56:50.627162 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:50.627104    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-79f457656b-dt4x8" podStartSLOduration=2.846865463 podStartE2EDuration="6.627089676s" podCreationTimestamp="2026-05-20 13:56:44 +0000 UTC" firstStartedPulling="2026-05-20 13:56:45.714776355 +0000 UTC m=+372.952392980" lastFinishedPulling="2026-05-20 13:56:49.495000568 +0000 UTC m=+376.732617193" observedRunningTime="2026-05-20 13:56:50.625746653 +0000 UTC m=+377.863363302" watchObservedRunningTime="2026-05-20 13:56:50.627089676 +0000 UTC m=+377.864706348"
May 20 13:56:52.616089 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:52.616054    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-68cc5db7c4-gh9gs" event={"ID":"2503b4b5-50c4-40d2-8e74-6de98c022740","Type":"ContainerStarted","Data":"25f8514946dc168cf35ad09231bcf593de3d32be56e1b076e09fca1511d71813"}
May 20 13:56:52.616477 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:52.616187    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-68cc5db7c4-gh9gs"
May 20 13:56:52.648352 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:52.648307    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-68cc5db7c4-gh9gs" podStartSLOduration=2.34154785 podStartE2EDuration="8.648292526s" podCreationTimestamp="2026-05-20 13:56:44 +0000 UTC" firstStartedPulling="2026-05-20 13:56:45.491896825 +0000 UTC m=+372.729513450" lastFinishedPulling="2026-05-20 13:56:51.798641498 +0000 UTC m=+379.036258126" observedRunningTime="2026-05-20 13:56:52.643766958 +0000 UTC m=+379.881383606" watchObservedRunningTime="2026-05-20 13:56:52.648292526 +0000 UTC m=+379.885909173"
May 20 13:56:56.613297 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:56:56.613264    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-79f457656b-dt4x8"
May 20 13:57:20.611993 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:20.611962    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-867cffc775-hv55f"
May 20 13:57:23.622334 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:23.622306    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-68cc5db7c4-gh9gs"
May 20 13:57:25.197286 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.197246    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-867cffc775-hv55f"]
May 20 13:57:25.197744 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.197509    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/kserve-controller-manager-867cffc775-hv55f" podUID="fb493e78-f437-47cb-b403-20e336a77c91" containerName="manager" containerID="cri-o://7666ecfe4508214fc3bbe193105cd2257b151ab5d02942ee5ecd7fd6757d3567" gracePeriod=10
May 20 13:57:25.259493 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.259456    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-867cffc775-hpvm4"]
May 20 13:57:25.261624 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.261598    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867cffc775-hpvm4"
May 20 13:57:25.279818 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.279787    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-867cffc775-hpvm4"]
May 20 13:57:25.385229 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.385199    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1125b1e5-8fbf-40e8-872b-61ce1ce07ec3-cert\") pod \"kserve-controller-manager-867cffc775-hpvm4\" (UID: \"1125b1e5-8fbf-40e8-872b-61ce1ce07ec3\") " pod="kserve/kserve-controller-manager-867cffc775-hpvm4"
May 20 13:57:25.385377 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.385266    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnlk8\" (UniqueName: \"kubernetes.io/projected/1125b1e5-8fbf-40e8-872b-61ce1ce07ec3-kube-api-access-qnlk8\") pod \"kserve-controller-manager-867cffc775-hpvm4\" (UID: \"1125b1e5-8fbf-40e8-872b-61ce1ce07ec3\") " pod="kserve/kserve-controller-manager-867cffc775-hpvm4"
May 20 13:57:25.430723 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.430701    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867cffc775-hv55f"
May 20 13:57:25.486132 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.486056    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsslt\" (UniqueName: \"kubernetes.io/projected/fb493e78-f437-47cb-b403-20e336a77c91-kube-api-access-wsslt\") pod \"fb493e78-f437-47cb-b403-20e336a77c91\" (UID: \"fb493e78-f437-47cb-b403-20e336a77c91\") "
May 20 13:57:25.486132 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.486102    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb493e78-f437-47cb-b403-20e336a77c91-cert\") pod \"fb493e78-f437-47cb-b403-20e336a77c91\" (UID: \"fb493e78-f437-47cb-b403-20e336a77c91\") "
May 20 13:57:25.486340 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.486187    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1125b1e5-8fbf-40e8-872b-61ce1ce07ec3-cert\") pod \"kserve-controller-manager-867cffc775-hpvm4\" (UID: \"1125b1e5-8fbf-40e8-872b-61ce1ce07ec3\") " pod="kserve/kserve-controller-manager-867cffc775-hpvm4"
May 20 13:57:25.486340 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.486252    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qnlk8\" (UniqueName: \"kubernetes.io/projected/1125b1e5-8fbf-40e8-872b-61ce1ce07ec3-kube-api-access-qnlk8\") pod \"kserve-controller-manager-867cffc775-hpvm4\" (UID: \"1125b1e5-8fbf-40e8-872b-61ce1ce07ec3\") " pod="kserve/kserve-controller-manager-867cffc775-hpvm4"
May 20 13:57:25.488973 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.488942    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb493e78-f437-47cb-b403-20e336a77c91-cert" (OuterVolumeSpecName: "cert") pod "fb493e78-f437-47cb-b403-20e336a77c91" (UID: "fb493e78-f437-47cb-b403-20e336a77c91"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:57:25.489107 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.489011    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb493e78-f437-47cb-b403-20e336a77c91-kube-api-access-wsslt" (OuterVolumeSpecName: "kube-api-access-wsslt") pod "fb493e78-f437-47cb-b403-20e336a77c91" (UID: "fb493e78-f437-47cb-b403-20e336a77c91"). InnerVolumeSpecName "kube-api-access-wsslt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:57:25.489107 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.489017    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1125b1e5-8fbf-40e8-872b-61ce1ce07ec3-cert\") pod \"kserve-controller-manager-867cffc775-hpvm4\" (UID: \"1125b1e5-8fbf-40e8-872b-61ce1ce07ec3\") " pod="kserve/kserve-controller-manager-867cffc775-hpvm4"
May 20 13:57:25.499679 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.499645    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnlk8\" (UniqueName: \"kubernetes.io/projected/1125b1e5-8fbf-40e8-872b-61ce1ce07ec3-kube-api-access-qnlk8\") pod \"kserve-controller-manager-867cffc775-hpvm4\" (UID: \"1125b1e5-8fbf-40e8-872b-61ce1ce07ec3\") " pod="kserve/kserve-controller-manager-867cffc775-hpvm4"
May 20 13:57:25.581122 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.581087    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867cffc775-hpvm4"
May 20 13:57:25.588101 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.588076    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsslt\" (UniqueName: \"kubernetes.io/projected/fb493e78-f437-47cb-b403-20e336a77c91-kube-api-access-wsslt\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:57:25.588101 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.588102    2570 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb493e78-f437-47cb-b403-20e336a77c91-cert\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:57:25.713881 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.713851    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-867cffc775-hpvm4"]
May 20 13:57:25.716431 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:57:25.716404    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1125b1e5_8fbf_40e8_872b_61ce1ce07ec3.slice/crio-5011689d12b61bbbf290d3d10765f6a186f85d5ae5eed977b4f62e2ac7a0846a WatchSource:0}: Error finding container 5011689d12b61bbbf290d3d10765f6a186f85d5ae5eed977b4f62e2ac7a0846a: Status 404 returned error can't find the container with id 5011689d12b61bbbf290d3d10765f6a186f85d5ae5eed977b4f62e2ac7a0846a
May 20 13:57:25.725271 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.725244    2570 generic.go:358] "Generic (PLEG): container finished" podID="fb493e78-f437-47cb-b403-20e336a77c91" containerID="7666ecfe4508214fc3bbe193105cd2257b151ab5d02942ee5ecd7fd6757d3567" exitCode=0
May 20 13:57:25.725366 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.725304    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867cffc775-hv55f"
May 20 13:57:25.725366 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.725322    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867cffc775-hv55f" event={"ID":"fb493e78-f437-47cb-b403-20e336a77c91","Type":"ContainerDied","Data":"7666ecfe4508214fc3bbe193105cd2257b151ab5d02942ee5ecd7fd6757d3567"}
May 20 13:57:25.725451 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.725369    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867cffc775-hv55f" event={"ID":"fb493e78-f437-47cb-b403-20e336a77c91","Type":"ContainerDied","Data":"659bf4311de984c5a4bb30f2f18690e0dbc887e2238dd83c20fe3f44ff8556af"}
May 20 13:57:25.725451 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.725394    2570 scope.go:117] "RemoveContainer" containerID="7666ecfe4508214fc3bbe193105cd2257b151ab5d02942ee5ecd7fd6757d3567"
May 20 13:57:25.726371 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.726353    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867cffc775-hpvm4" event={"ID":"1125b1e5-8fbf-40e8-872b-61ce1ce07ec3","Type":"ContainerStarted","Data":"5011689d12b61bbbf290d3d10765f6a186f85d5ae5eed977b4f62e2ac7a0846a"}
May 20 13:57:25.733982 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.733826    2570 scope.go:117] "RemoveContainer" containerID="7666ecfe4508214fc3bbe193105cd2257b151ab5d02942ee5ecd7fd6757d3567"
May 20 13:57:25.734147 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:57:25.734112    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7666ecfe4508214fc3bbe193105cd2257b151ab5d02942ee5ecd7fd6757d3567\": container with ID starting with 7666ecfe4508214fc3bbe193105cd2257b151ab5d02942ee5ecd7fd6757d3567 not found: ID does not exist" containerID="7666ecfe4508214fc3bbe193105cd2257b151ab5d02942ee5ecd7fd6757d3567"
May 20 13:57:25.734216 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.734149    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7666ecfe4508214fc3bbe193105cd2257b151ab5d02942ee5ecd7fd6757d3567"} err="failed to get container status \"7666ecfe4508214fc3bbe193105cd2257b151ab5d02942ee5ecd7fd6757d3567\": rpc error: code = NotFound desc = could not find container \"7666ecfe4508214fc3bbe193105cd2257b151ab5d02942ee5ecd7fd6757d3567\": container with ID starting with 7666ecfe4508214fc3bbe193105cd2257b151ab5d02942ee5ecd7fd6757d3567 not found: ID does not exist"
May 20 13:57:25.751103 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.751023    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-867cffc775-hv55f"]
May 20 13:57:25.755877 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:25.755838    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-867cffc775-hv55f"]
May 20 13:57:26.731474 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:26.731438    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867cffc775-hpvm4" event={"ID":"1125b1e5-8fbf-40e8-872b-61ce1ce07ec3","Type":"ContainerStarted","Data":"ae90acd71afbf01e26435a9c949e743c5c074242063e5fc010fc98b567aea156"}
May 20 13:57:26.731877 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:26.731529    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-867cffc775-hpvm4"
May 20 13:57:26.757252 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:26.757034    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-867cffc775-hpvm4" podStartSLOduration=1.3574612400000001 podStartE2EDuration="1.757016316s" podCreationTimestamp="2026-05-20 13:57:25 +0000 UTC" firstStartedPulling="2026-05-20 13:57:25.717662368 +0000 UTC m=+412.955278996" lastFinishedPulling="2026-05-20 13:57:26.117217434 +0000 UTC m=+413.354834072" observedRunningTime="2026-05-20 13:57:26.756951395 +0000 UTC m=+413.994568052" watchObservedRunningTime="2026-05-20 13:57:26.757016316 +0000 UTC m=+413.994632963"
May 20 13:57:27.364692 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:27.364659    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb493e78-f437-47cb-b403-20e336a77c91" path="/var/lib/kubelet/pods/fb493e78-f437-47cb-b403-20e336a77c91/volumes"
May 20 13:57:57.740466 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:57.740387    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-867cffc775-hpvm4"
May 20 13:57:58.944822 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:58.944785    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-q46ld"]
May 20 13:57:58.945198 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:58.945127    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb493e78-f437-47cb-b403-20e336a77c91" containerName="manager"
May 20 13:57:58.945198 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:58.945138    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb493e78-f437-47cb-b403-20e336a77c91" containerName="manager"
May 20 13:57:58.945198 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:58.945192    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="fb493e78-f437-47cb-b403-20e336a77c91" containerName="manager"
May 20 13:57:58.948122 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:58.948094    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-q46ld"
May 20 13:57:58.953067 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:58.953043    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-6lh96\""
May 20 13:57:58.953067 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:58.953045    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
May 20 13:57:58.959706 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:58.959680    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-cp94v"]
May 20 13:57:58.962752 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:58.962733    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-cp94v"
May 20 13:57:58.971033 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:58.971007    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
May 20 13:57:58.971033 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:58.971016    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-w5zdt\""
May 20 13:57:58.971673 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:58.971652    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-q46ld"]
May 20 13:57:58.972523 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:58.972507    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fef87a76-8df3-43e2-af68-0ec0f069a524-cert\") pod \"odh-model-controller-696fc77849-cp94v\" (UID: \"fef87a76-8df3-43e2-af68-0ec0f069a524\") " pod="kserve/odh-model-controller-696fc77849-cp94v"
May 20 13:57:58.972588 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:58.972534    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f907942-12ba-46b9-979a-90c37b400ca0-tls-certs\") pod \"model-serving-api-86f7b4b499-q46ld\" (UID: \"5f907942-12ba-46b9-979a-90c37b400ca0\") " pod="kserve/model-serving-api-86f7b4b499-q46ld"
May 20 13:57:58.972588 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:58.972551    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdrf9\" (UniqueName: \"kubernetes.io/projected/5f907942-12ba-46b9-979a-90c37b400ca0-kube-api-access-rdrf9\") pod \"model-serving-api-86f7b4b499-q46ld\" (UID: \"5f907942-12ba-46b9-979a-90c37b400ca0\") " pod="kserve/model-serving-api-86f7b4b499-q46ld"
May 20 13:57:58.972588 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:58.972575    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfk9p\" (UniqueName: \"kubernetes.io/projected/fef87a76-8df3-43e2-af68-0ec0f069a524-kube-api-access-pfk9p\") pod \"odh-model-controller-696fc77849-cp94v\" (UID: \"fef87a76-8df3-43e2-af68-0ec0f069a524\") " pod="kserve/odh-model-controller-696fc77849-cp94v"
May 20 13:57:58.982901 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:58.982878    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-cp94v"]
May 20 13:57:59.073138 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:59.073106    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fef87a76-8df3-43e2-af68-0ec0f069a524-cert\") pod \"odh-model-controller-696fc77849-cp94v\" (UID: \"fef87a76-8df3-43e2-af68-0ec0f069a524\") " pod="kserve/odh-model-controller-696fc77849-cp94v"
May 20 13:57:59.073138 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:59.073139    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f907942-12ba-46b9-979a-90c37b400ca0-tls-certs\") pod \"model-serving-api-86f7b4b499-q46ld\" (UID: \"5f907942-12ba-46b9-979a-90c37b400ca0\") " pod="kserve/model-serving-api-86f7b4b499-q46ld"
May 20 13:57:59.073348 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:59.073170    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rdrf9\" (UniqueName: \"kubernetes.io/projected/5f907942-12ba-46b9-979a-90c37b400ca0-kube-api-access-rdrf9\") pod \"model-serving-api-86f7b4b499-q46ld\" (UID: \"5f907942-12ba-46b9-979a-90c37b400ca0\") " pod="kserve/model-serving-api-86f7b4b499-q46ld"
May 20 13:57:59.073348 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:59.073198    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pfk9p\" (UniqueName: \"kubernetes.io/projected/fef87a76-8df3-43e2-af68-0ec0f069a524-kube-api-access-pfk9p\") pod \"odh-model-controller-696fc77849-cp94v\" (UID: \"fef87a76-8df3-43e2-af68-0ec0f069a524\") " pod="kserve/odh-model-controller-696fc77849-cp94v"
May 20 13:57:59.075746 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:59.075724    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fef87a76-8df3-43e2-af68-0ec0f069a524-cert\") pod \"odh-model-controller-696fc77849-cp94v\" (UID: \"fef87a76-8df3-43e2-af68-0ec0f069a524\") " pod="kserve/odh-model-controller-696fc77849-cp94v"
May 20 13:57:59.075867 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:59.075779    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f907942-12ba-46b9-979a-90c37b400ca0-tls-certs\") pod \"model-serving-api-86f7b4b499-q46ld\" (UID: \"5f907942-12ba-46b9-979a-90c37b400ca0\") " pod="kserve/model-serving-api-86f7b4b499-q46ld"
May 20 13:57:59.089777 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:59.089752    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfk9p\" (UniqueName: \"kubernetes.io/projected/fef87a76-8df3-43e2-af68-0ec0f069a524-kube-api-access-pfk9p\") pod \"odh-model-controller-696fc77849-cp94v\" (UID: \"fef87a76-8df3-43e2-af68-0ec0f069a524\") " pod="kserve/odh-model-controller-696fc77849-cp94v"
May 20 13:57:59.089912 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:59.089847    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdrf9\" (UniqueName: \"kubernetes.io/projected/5f907942-12ba-46b9-979a-90c37b400ca0-kube-api-access-rdrf9\") pod \"model-serving-api-86f7b4b499-q46ld\" (UID: \"5f907942-12ba-46b9-979a-90c37b400ca0\") " pod="kserve/model-serving-api-86f7b4b499-q46ld"
May 20 13:57:59.258669 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:59.258561    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-q46ld"
May 20 13:57:59.272388 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:59.272355    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-cp94v"
May 20 13:57:59.407454 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:59.407355    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-q46ld"]
May 20 13:57:59.409744 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:57:59.409713    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f907942_12ba_46b9_979a_90c37b400ca0.slice/crio-0b10b5c43edf2d6e31a4870d25b256d7d15e0cbf7bda3a90b976d30b7e510662 WatchSource:0}: Error finding container 0b10b5c43edf2d6e31a4870d25b256d7d15e0cbf7bda3a90b976d30b7e510662: Status 404 returned error can't find the container with id 0b10b5c43edf2d6e31a4870d25b256d7d15e0cbf7bda3a90b976d30b7e510662
May 20 13:57:59.432050 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:59.432018    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-cp94v"]
May 20 13:57:59.434811 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:57:59.434783    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef87a76_8df3_43e2_af68_0ec0f069a524.slice/crio-0837bd4e2a069a45ed97c153128991d98057a7e52cc57c65c5628006ae8720b2 WatchSource:0}: Error finding container 0837bd4e2a069a45ed97c153128991d98057a7e52cc57c65c5628006ae8720b2: Status 404 returned error can't find the container with id 0837bd4e2a069a45ed97c153128991d98057a7e52cc57c65c5628006ae8720b2
May 20 13:57:59.839379 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:59.839344    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-cp94v" event={"ID":"fef87a76-8df3-43e2-af68-0ec0f069a524","Type":"ContainerStarted","Data":"0837bd4e2a069a45ed97c153128991d98057a7e52cc57c65c5628006ae8720b2"}
May 20 13:57:59.840267 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:57:59.840241    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-q46ld" event={"ID":"5f907942-12ba-46b9-979a-90c37b400ca0","Type":"ContainerStarted","Data":"0b10b5c43edf2d6e31a4870d25b256d7d15e0cbf7bda3a90b976d30b7e510662"}
May 20 13:58:03.856347 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:03.856304    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-q46ld" event={"ID":"5f907942-12ba-46b9-979a-90c37b400ca0","Type":"ContainerStarted","Data":"60fd7d730fcb1a098c8d301a22c67093226069c4ec99a4e75c79bb79f54686a9"}
May 20 13:58:03.856883 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:03.856658    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-q46ld"
May 20 13:58:03.857897 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:03.857873    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-cp94v" event={"ID":"fef87a76-8df3-43e2-af68-0ec0f069a524","Type":"ContainerStarted","Data":"d99cb80ebf067f67104d22146479c035394b91696fa1d62261b5ffdba7f8bbef"}
May 20 13:58:03.858019 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:03.858000    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-cp94v"
May 20 13:58:03.898944 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:03.898882    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-q46ld" podStartSLOduration=2.176907557 podStartE2EDuration="5.898862677s" podCreationTimestamp="2026-05-20 13:57:58 +0000 UTC" firstStartedPulling="2026-05-20 13:57:59.411520821 +0000 UTC m=+446.649137447" lastFinishedPulling="2026-05-20 13:58:03.133475927 +0000 UTC m=+450.371092567" observedRunningTime="2026-05-20 13:58:03.895273809 +0000 UTC m=+451.132890480" watchObservedRunningTime="2026-05-20 13:58:03.898862677 +0000 UTC m=+451.136479325"
May 20 13:58:03.922483 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:03.922430    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-cp94v" podStartSLOduration=2.220526738 podStartE2EDuration="5.922414793s" podCreationTimestamp="2026-05-20 13:57:58 +0000 UTC" firstStartedPulling="2026-05-20 13:57:59.436024772 +0000 UTC m=+446.673641397" lastFinishedPulling="2026-05-20 13:58:03.137912822 +0000 UTC m=+450.375529452" observedRunningTime="2026-05-20 13:58:03.921985692 +0000 UTC m=+451.159602345" watchObservedRunningTime="2026-05-20 13:58:03.922414793 +0000 UTC m=+451.160031440"
May 20 13:58:04.999121 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:04.999062    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f5b9cdfc5-hh7lz"]
May 20 13:58:05.002715 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.002694    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.017767 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.017742    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f5b9cdfc5-hh7lz"]
May 20 13:58:05.023733 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.023709    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f69e3276-e4a9-47fa-a1db-e18f14eba62b-trusted-ca-bundle\") pod \"console-5f5b9cdfc5-hh7lz\" (UID: \"f69e3276-e4a9-47fa-a1db-e18f14eba62b\") " pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.023843 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.023739    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvp7c\" (UniqueName: \"kubernetes.io/projected/f69e3276-e4a9-47fa-a1db-e18f14eba62b-kube-api-access-gvp7c\") pod \"console-5f5b9cdfc5-hh7lz\" (UID: \"f69e3276-e4a9-47fa-a1db-e18f14eba62b\") " pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.023843 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.023809    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f69e3276-e4a9-47fa-a1db-e18f14eba62b-console-serving-cert\") pod \"console-5f5b9cdfc5-hh7lz\" (UID: \"f69e3276-e4a9-47fa-a1db-e18f14eba62b\") " pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.023843 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.023834    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f69e3276-e4a9-47fa-a1db-e18f14eba62b-console-config\") pod \"console-5f5b9cdfc5-hh7lz\" (UID: \"f69e3276-e4a9-47fa-a1db-e18f14eba62b\") " pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.023981 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.023906    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f69e3276-e4a9-47fa-a1db-e18f14eba62b-console-oauth-config\") pod \"console-5f5b9cdfc5-hh7lz\" (UID: \"f69e3276-e4a9-47fa-a1db-e18f14eba62b\") " pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.023981 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.023940    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f69e3276-e4a9-47fa-a1db-e18f14eba62b-oauth-serving-cert\") pod \"console-5f5b9cdfc5-hh7lz\" (UID: \"f69e3276-e4a9-47fa-a1db-e18f14eba62b\") " pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.023981 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.023963    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f69e3276-e4a9-47fa-a1db-e18f14eba62b-service-ca\") pod \"console-5f5b9cdfc5-hh7lz\" (UID: \"f69e3276-e4a9-47fa-a1db-e18f14eba62b\") " pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.124419 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.124380    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f69e3276-e4a9-47fa-a1db-e18f14eba62b-console-oauth-config\") pod \"console-5f5b9cdfc5-hh7lz\" (UID: \"f69e3276-e4a9-47fa-a1db-e18f14eba62b\") " pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.124419 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.124425    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f69e3276-e4a9-47fa-a1db-e18f14eba62b-oauth-serving-cert\") pod \"console-5f5b9cdfc5-hh7lz\" (UID: \"f69e3276-e4a9-47fa-a1db-e18f14eba62b\") " pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.124715 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.124458    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f69e3276-e4a9-47fa-a1db-e18f14eba62b-service-ca\") pod \"console-5f5b9cdfc5-hh7lz\" (UID: \"f69e3276-e4a9-47fa-a1db-e18f14eba62b\") " pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.124715 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.124485    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f69e3276-e4a9-47fa-a1db-e18f14eba62b-trusted-ca-bundle\") pod \"console-5f5b9cdfc5-hh7lz\" (UID: \"f69e3276-e4a9-47fa-a1db-e18f14eba62b\") " pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.124715 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.124512    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gvp7c\" (UniqueName: \"kubernetes.io/projected/f69e3276-e4a9-47fa-a1db-e18f14eba62b-kube-api-access-gvp7c\") pod \"console-5f5b9cdfc5-hh7lz\" (UID: \"f69e3276-e4a9-47fa-a1db-e18f14eba62b\") " pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.124715 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.124580    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f69e3276-e4a9-47fa-a1db-e18f14eba62b-console-serving-cert\") pod \"console-5f5b9cdfc5-hh7lz\" (UID: \"f69e3276-e4a9-47fa-a1db-e18f14eba62b\") " pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.124715 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.124602    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f69e3276-e4a9-47fa-a1db-e18f14eba62b-console-config\") pod \"console-5f5b9cdfc5-hh7lz\" (UID: \"f69e3276-e4a9-47fa-a1db-e18f14eba62b\") " pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.125311 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.125282    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f69e3276-e4a9-47fa-a1db-e18f14eba62b-oauth-serving-cert\") pod \"console-5f5b9cdfc5-hh7lz\" (UID: \"f69e3276-e4a9-47fa-a1db-e18f14eba62b\") " pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.125480 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.125457    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f69e3276-e4a9-47fa-a1db-e18f14eba62b-trusted-ca-bundle\") pod \"console-5f5b9cdfc5-hh7lz\" (UID: \"f69e3276-e4a9-47fa-a1db-e18f14eba62b\") " pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.125544 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.125485    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f69e3276-e4a9-47fa-a1db-e18f14eba62b-console-config\") pod \"console-5f5b9cdfc5-hh7lz\" (UID: \"f69e3276-e4a9-47fa-a1db-e18f14eba62b\") " pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.125744 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.125721    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f69e3276-e4a9-47fa-a1db-e18f14eba62b-service-ca\") pod \"console-5f5b9cdfc5-hh7lz\" (UID: \"f69e3276-e4a9-47fa-a1db-e18f14eba62b\") " pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.127217 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.127189    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f69e3276-e4a9-47fa-a1db-e18f14eba62b-console-oauth-config\") pod \"console-5f5b9cdfc5-hh7lz\" (UID: \"f69e3276-e4a9-47fa-a1db-e18f14eba62b\") " pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.127310 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.127282    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f69e3276-e4a9-47fa-a1db-e18f14eba62b-console-serving-cert\") pod \"console-5f5b9cdfc5-hh7lz\" (UID: \"f69e3276-e4a9-47fa-a1db-e18f14eba62b\") " pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.133337 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.133306    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvp7c\" (UniqueName: \"kubernetes.io/projected/f69e3276-e4a9-47fa-a1db-e18f14eba62b-kube-api-access-gvp7c\") pod \"console-5f5b9cdfc5-hh7lz\" (UID: \"f69e3276-e4a9-47fa-a1db-e18f14eba62b\") " pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.312462 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.312428    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:05.459952 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.459923    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f5b9cdfc5-hh7lz"]
May 20 13:58:05.462310 ip-10-0-136-75 kubenswrapper[2570]: W0520 13:58:05.462284    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69e3276_e4a9_47fa_a1db_e18f14eba62b.slice/crio-61f92b64fffc48b80eaca9be7f639f5a17c9b97a6986d3e0273b39385f255ee5 WatchSource:0}: Error finding container 61f92b64fffc48b80eaca9be7f639f5a17c9b97a6986d3e0273b39385f255ee5: Status 404 returned error can't find the container with id 61f92b64fffc48b80eaca9be7f639f5a17c9b97a6986d3e0273b39385f255ee5
May 20 13:58:05.865931 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.865892    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f5b9cdfc5-hh7lz" event={"ID":"f69e3276-e4a9-47fa-a1db-e18f14eba62b","Type":"ContainerStarted","Data":"4aeee6dfb6d60f4627056d3d8284cc480d3d42d45eb11181cd59f60b13091714"}
May 20 13:58:05.866114 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.865937    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f5b9cdfc5-hh7lz" event={"ID":"f69e3276-e4a9-47fa-a1db-e18f14eba62b","Type":"ContainerStarted","Data":"61f92b64fffc48b80eaca9be7f639f5a17c9b97a6986d3e0273b39385f255ee5"}
May 20 13:58:05.895294 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:05.895243    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5f5b9cdfc5-hh7lz" podStartSLOduration=1.8952296629999998 podStartE2EDuration="1.895229663s" podCreationTimestamp="2026-05-20 13:58:04 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:58:05.894405165 +0000 UTC m=+453.132021812" watchObservedRunningTime="2026-05-20 13:58:05.895229663 +0000 UTC m=+453.132846311"
May 20 13:58:14.864335 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:14.864305    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-cp94v"
May 20 13:58:14.870063 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:14.866872    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-q46ld"
May 20 13:58:15.313049 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:15.313021    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:15.313226 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:15.313057    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:15.317691 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:15.317671    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:15.904725 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:15.904697    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f5b9cdfc5-hh7lz"
May 20 13:58:15.969592 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:15.969558    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-665cbfb858-k9mf7"]
May 20 13:58:40.989876 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:40.989818    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-665cbfb858-k9mf7" podUID="d81baafa-32b8-4152-b4bd-f727b0f6a18a" containerName="console" containerID="cri-o://d6c48ffbf24292631089d2b505608ed6fb84bb5e29eb8f7d860801afdb147957" gracePeriod=15
May 20 13:58:41.231097 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.231074    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-665cbfb858-k9mf7_d81baafa-32b8-4152-b4bd-f727b0f6a18a/console/0.log"
May 20 13:58:41.231229 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.231140    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:58:41.348448 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.348413    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-trusted-ca-bundle\") pod \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") "
May 20 13:58:41.348650 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.348468    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-service-ca\") pod \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") "
May 20 13:58:41.348650 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.348506    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fn5l\" (UniqueName: \"kubernetes.io/projected/d81baafa-32b8-4152-b4bd-f727b0f6a18a-kube-api-access-4fn5l\") pod \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") "
May 20 13:58:41.348737 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.348670    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d81baafa-32b8-4152-b4bd-f727b0f6a18a-console-serving-cert\") pod \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") "
May 20 13:58:41.348796 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.348765    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-oauth-serving-cert\") pod \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") "
May 20 13:58:41.348854 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.348796    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-console-config\") pod \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") "
May 20 13:58:41.348854 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.348828    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d81baafa-32b8-4152-b4bd-f727b0f6a18a-console-oauth-config\") pod \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\" (UID: \"d81baafa-32b8-4152-b4bd-f727b0f6a18a\") "
May 20 13:58:41.348951 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.348877    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d81baafa-32b8-4152-b4bd-f727b0f6a18a" (UID: "d81baafa-32b8-4152-b4bd-f727b0f6a18a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:58:41.349098 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.349075    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-service-ca" (OuterVolumeSpecName: "service-ca") pod "d81baafa-32b8-4152-b4bd-f727b0f6a18a" (UID: "d81baafa-32b8-4152-b4bd-f727b0f6a18a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:58:41.349170 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.349106    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-trusted-ca-bundle\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:58:41.349170 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.349140    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d81baafa-32b8-4152-b4bd-f727b0f6a18a" (UID: "d81baafa-32b8-4152-b4bd-f727b0f6a18a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:58:41.349170 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.349144    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-console-config" (OuterVolumeSpecName: "console-config") pod "d81baafa-32b8-4152-b4bd-f727b0f6a18a" (UID: "d81baafa-32b8-4152-b4bd-f727b0f6a18a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:58:41.351026 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.350995    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81baafa-32b8-4152-b4bd-f727b0f6a18a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d81baafa-32b8-4152-b4bd-f727b0f6a18a" (UID: "d81baafa-32b8-4152-b4bd-f727b0f6a18a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:58:41.351155 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.351031    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81baafa-32b8-4152-b4bd-f727b0f6a18a-kube-api-access-4fn5l" (OuterVolumeSpecName: "kube-api-access-4fn5l") pod "d81baafa-32b8-4152-b4bd-f727b0f6a18a" (UID: "d81baafa-32b8-4152-b4bd-f727b0f6a18a"). InnerVolumeSpecName "kube-api-access-4fn5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:58:41.351155 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.351126    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81baafa-32b8-4152-b4bd-f727b0f6a18a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d81baafa-32b8-4152-b4bd-f727b0f6a18a" (UID: "d81baafa-32b8-4152-b4bd-f727b0f6a18a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:58:41.450421 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.450384    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-service-ca\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:58:41.450421 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.450415    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fn5l\" (UniqueName: \"kubernetes.io/projected/d81baafa-32b8-4152-b4bd-f727b0f6a18a-kube-api-access-4fn5l\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:58:41.450421 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.450430    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d81baafa-32b8-4152-b4bd-f727b0f6a18a-console-serving-cert\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:58:41.450695 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.450439    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-oauth-serving-cert\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:58:41.450695 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.450448    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d81baafa-32b8-4152-b4bd-f727b0f6a18a-console-config\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:58:41.450695 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.450457    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d81baafa-32b8-4152-b4bd-f727b0f6a18a-console-oauth-config\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 13:58:41.989566 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.989537    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-665cbfb858-k9mf7_d81baafa-32b8-4152-b4bd-f727b0f6a18a/console/0.log"
May 20 13:58:41.989755 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.989581    2570 generic.go:358] "Generic (PLEG): container finished" podID="d81baafa-32b8-4152-b4bd-f727b0f6a18a" containerID="d6c48ffbf24292631089d2b505608ed6fb84bb5e29eb8f7d860801afdb147957" exitCode=2
May 20 13:58:41.989755 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.989668    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-665cbfb858-k9mf7" event={"ID":"d81baafa-32b8-4152-b4bd-f727b0f6a18a","Type":"ContainerDied","Data":"d6c48ffbf24292631089d2b505608ed6fb84bb5e29eb8f7d860801afdb147957"}
May 20 13:58:41.989755 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.989696    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-665cbfb858-k9mf7" event={"ID":"d81baafa-32b8-4152-b4bd-f727b0f6a18a","Type":"ContainerDied","Data":"7bf9005d8d92695240f85d91876cf56248f89ae48775a8a1f6e94c86dccbba0e"}
May 20 13:58:41.989755 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.989710    2570 scope.go:117] "RemoveContainer" containerID="d6c48ffbf24292631089d2b505608ed6fb84bb5e29eb8f7d860801afdb147957"
May 20 13:58:41.989909 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.989707    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-665cbfb858-k9mf7"
May 20 13:58:41.997653 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.997631    2570 scope.go:117] "RemoveContainer" containerID="d6c48ffbf24292631089d2b505608ed6fb84bb5e29eb8f7d860801afdb147957"
May 20 13:58:41.997908 ip-10-0-136-75 kubenswrapper[2570]: E0520 13:58:41.997886    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6c48ffbf24292631089d2b505608ed6fb84bb5e29eb8f7d860801afdb147957\": container with ID starting with d6c48ffbf24292631089d2b505608ed6fb84bb5e29eb8f7d860801afdb147957 not found: ID does not exist" containerID="d6c48ffbf24292631089d2b505608ed6fb84bb5e29eb8f7d860801afdb147957"
May 20 13:58:41.997979 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:41.997921    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c48ffbf24292631089d2b505608ed6fb84bb5e29eb8f7d860801afdb147957"} err="failed to get container status \"d6c48ffbf24292631089d2b505608ed6fb84bb5e29eb8f7d860801afdb147957\": rpc error: code = NotFound desc = could not find container \"d6c48ffbf24292631089d2b505608ed6fb84bb5e29eb8f7d860801afdb147957\": container with ID starting with d6c48ffbf24292631089d2b505608ed6fb84bb5e29eb8f7d860801afdb147957 not found: ID does not exist"
May 20 13:58:42.012062 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:42.012036    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-665cbfb858-k9mf7"]
May 20 13:58:42.017709 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:42.017681    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-665cbfb858-k9mf7"]
May 20 13:58:43.363803 ip-10-0-136-75 kubenswrapper[2570]: I0520 13:58:43.363767    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d81baafa-32b8-4152-b4bd-f727b0f6a18a" path="/var/lib/kubelet/pods/d81baafa-32b8-4152-b4bd-f727b0f6a18a/volumes"
May 20 14:00:33.277412 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:00:33.277380    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 14:00:33.279233 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:00:33.279208    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 14:01:38.500748 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.500708    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"]
May 20 14:01:38.501179 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.501053    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81baafa-32b8-4152-b4bd-f727b0f6a18a" containerName="console"
May 20 14:01:38.501179 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.501064    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81baafa-32b8-4152-b4bd-f727b0f6a18a" containerName="console"
May 20 14:01:38.501179 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.501129    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="d81baafa-32b8-4152-b4bd-f727b0f6a18a" containerName="console"
May 20 14:01:38.503245 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.503226    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"
May 20 14:01:38.509223 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.509194    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-69925-kube-rbac-proxy-sar-config\""
May 20 14:01:38.509390 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.509235    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 14:01:38.509390 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.509245    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-k2v42\""
May 20 14:01:38.509390 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.509285    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-69925-predictor-serving-cert\""
May 20 14:01:38.509390 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.509247    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 14:01:38.517569 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.517535    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"]
May 20 14:01:38.565156 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.565115    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b977c2d-8ee4-4db5-bb33-e828b9e1db25-proxy-tls\") pod \"error-404-isvc-69925-predictor-5d57446965-8zjs4\" (UID: \"3b977c2d-8ee4-4db5-bb33-e828b9e1db25\") " pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"
May 20 14:01:38.565315 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.565177    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-69925-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b977c2d-8ee4-4db5-bb33-e828b9e1db25-error-404-isvc-69925-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-69925-predictor-5d57446965-8zjs4\" (UID: \"3b977c2d-8ee4-4db5-bb33-e828b9e1db25\") " pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"
May 20 14:01:38.565315 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.565243    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xvw7\" (UniqueName: \"kubernetes.io/projected/3b977c2d-8ee4-4db5-bb33-e828b9e1db25-kube-api-access-8xvw7\") pod \"error-404-isvc-69925-predictor-5d57446965-8zjs4\" (UID: \"3b977c2d-8ee4-4db5-bb33-e828b9e1db25\") " pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"
May 20 14:01:38.666074 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.666031    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-69925-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b977c2d-8ee4-4db5-bb33-e828b9e1db25-error-404-isvc-69925-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-69925-predictor-5d57446965-8zjs4\" (UID: \"3b977c2d-8ee4-4db5-bb33-e828b9e1db25\") " pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"
May 20 14:01:38.666306 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.666212    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8xvw7\" (UniqueName: \"kubernetes.io/projected/3b977c2d-8ee4-4db5-bb33-e828b9e1db25-kube-api-access-8xvw7\") pod \"error-404-isvc-69925-predictor-5d57446965-8zjs4\" (UID: \"3b977c2d-8ee4-4db5-bb33-e828b9e1db25\") " pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"
May 20 14:01:38.666306 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.666289    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b977c2d-8ee4-4db5-bb33-e828b9e1db25-proxy-tls\") pod \"error-404-isvc-69925-predictor-5d57446965-8zjs4\" (UID: \"3b977c2d-8ee4-4db5-bb33-e828b9e1db25\") " pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"
May 20 14:01:38.666769 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.666745    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-69925-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b977c2d-8ee4-4db5-bb33-e828b9e1db25-error-404-isvc-69925-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-69925-predictor-5d57446965-8zjs4\" (UID: \"3b977c2d-8ee4-4db5-bb33-e828b9e1db25\") " pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"
May 20 14:01:38.669118 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.669085    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b977c2d-8ee4-4db5-bb33-e828b9e1db25-proxy-tls\") pod \"error-404-isvc-69925-predictor-5d57446965-8zjs4\" (UID: \"3b977c2d-8ee4-4db5-bb33-e828b9e1db25\") " pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"
May 20 14:01:38.677820 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.677785    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xvw7\" (UniqueName: \"kubernetes.io/projected/3b977c2d-8ee4-4db5-bb33-e828b9e1db25-kube-api-access-8xvw7\") pod \"error-404-isvc-69925-predictor-5d57446965-8zjs4\" (UID: \"3b977c2d-8ee4-4db5-bb33-e828b9e1db25\") " pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"
May 20 14:01:38.815585 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.815538    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"
May 20 14:01:38.961783 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.961748    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"]
May 20 14:01:38.965120 ip-10-0-136-75 kubenswrapper[2570]: W0520 14:01:38.965089    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b977c2d_8ee4_4db5_bb33_e828b9e1db25.slice/crio-e93f17c79288a780ce72e8deaa838ff6df2ebf06735faa1448b46aa25aa0f13f WatchSource:0}: Error finding container e93f17c79288a780ce72e8deaa838ff6df2ebf06735faa1448b46aa25aa0f13f: Status 404 returned error can't find the container with id e93f17c79288a780ce72e8deaa838ff6df2ebf06735faa1448b46aa25aa0f13f
May 20 14:01:38.966991 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:38.966966    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:01:39.585509 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:39.585465    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4" event={"ID":"3b977c2d-8ee4-4db5-bb33-e828b9e1db25","Type":"ContainerStarted","Data":"e93f17c79288a780ce72e8deaa838ff6df2ebf06735faa1448b46aa25aa0f13f"}
May 20 14:01:50.650291 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:50.650194    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4" event={"ID":"3b977c2d-8ee4-4db5-bb33-e828b9e1db25","Type":"ContainerStarted","Data":"49a1592c07cdda4b3740a8e11157408e68813a3caed5ea4e26e09472c4ece120"}
May 20 14:01:52.658144 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:52.658106    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4" event={"ID":"3b977c2d-8ee4-4db5-bb33-e828b9e1db25","Type":"ContainerStarted","Data":"293c1d52e50fa022f9006747b2f5e2391f93c4ceb8e1a3f76ddc6042660ac557"}
May 20 14:01:52.658555 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:52.658312    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"
May 20 14:01:52.658555 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:52.658420    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"
May 20 14:01:52.659695 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:52.659661    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4" podUID="3b977c2d-8ee4-4db5-bb33-e828b9e1db25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 20 14:01:52.686344 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:52.686292    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4" podStartSLOduration=1.112531338 podStartE2EDuration="14.686260925s" podCreationTimestamp="2026-05-20 14:01:38 +0000 UTC" firstStartedPulling="2026-05-20 14:01:38.967120871 +0000 UTC m=+666.204737499" lastFinishedPulling="2026-05-20 14:01:52.540850447 +0000 UTC m=+679.778467086" observedRunningTime="2026-05-20 14:01:52.684049862 +0000 UTC m=+679.921666509" watchObservedRunningTime="2026-05-20 14:01:52.686260925 +0000 UTC m=+679.923877571"
May 20 14:01:53.661595 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:53.661553    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4" podUID="3b977c2d-8ee4-4db5-bb33-e828b9e1db25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 20 14:01:58.666705 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:58.666663    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"
May 20 14:01:58.667221 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:01:58.667153    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4" podUID="3b977c2d-8ee4-4db5-bb33-e828b9e1db25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 20 14:02:08.667539 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:02:08.667488    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4" podUID="3b977c2d-8ee4-4db5-bb33-e828b9e1db25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 20 14:02:18.667680 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:02:18.667638    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4" podUID="3b977c2d-8ee4-4db5-bb33-e828b9e1db25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 20 14:02:28.667714 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:02:28.667668    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4" podUID="3b977c2d-8ee4-4db5-bb33-e828b9e1db25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 20 14:02:38.667779 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:02:38.667744    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"
May 20 14:03:08.252328 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:08.252292    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6"]
May 20 14:03:08.254718 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:08.254701    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6"
May 20 14:03:08.257740 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:08.257718    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-69925-serving-cert\""
May 20 14:03:08.257963 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:08.257949    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-69925-kube-rbac-proxy-sar-config\""
May 20 14:03:08.266053 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:08.266024    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6"]
May 20 14:03:08.351829 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:08.351792    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccbd9ee3-f4d3-4d14-970a-257df2ffb815-openshift-service-ca-bundle\") pod \"switch-graph-69925-7c7b5bdf4d-g6qn6\" (UID: \"ccbd9ee3-f4d3-4d14-970a-257df2ffb815\") " pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6"
May 20 14:03:08.351829 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:08.351832    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccbd9ee3-f4d3-4d14-970a-257df2ffb815-proxy-tls\") pod \"switch-graph-69925-7c7b5bdf4d-g6qn6\" (UID: \"ccbd9ee3-f4d3-4d14-970a-257df2ffb815\") " pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6"
May 20 14:03:08.452827 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:08.452792    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccbd9ee3-f4d3-4d14-970a-257df2ffb815-openshift-service-ca-bundle\") pod \"switch-graph-69925-7c7b5bdf4d-g6qn6\" (UID: \"ccbd9ee3-f4d3-4d14-970a-257df2ffb815\") " pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6"
May 20 14:03:08.453004 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:08.452835    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccbd9ee3-f4d3-4d14-970a-257df2ffb815-proxy-tls\") pod \"switch-graph-69925-7c7b5bdf4d-g6qn6\" (UID: \"ccbd9ee3-f4d3-4d14-970a-257df2ffb815\") " pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6"
May 20 14:03:08.453004 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:03:08.452963    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-69925-serving-cert: secret "switch-graph-69925-serving-cert" not found
May 20 14:03:08.453074 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:03:08.453033    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccbd9ee3-f4d3-4d14-970a-257df2ffb815-proxy-tls podName:ccbd9ee3-f4d3-4d14-970a-257df2ffb815 nodeName:}" failed. No retries permitted until 2026-05-20 14:03:08.953015249 +0000 UTC m=+756.190631874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ccbd9ee3-f4d3-4d14-970a-257df2ffb815-proxy-tls") pod "switch-graph-69925-7c7b5bdf4d-g6qn6" (UID: "ccbd9ee3-f4d3-4d14-970a-257df2ffb815") : secret "switch-graph-69925-serving-cert" not found
May 20 14:03:08.453407 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:08.453389    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccbd9ee3-f4d3-4d14-970a-257df2ffb815-openshift-service-ca-bundle\") pod \"switch-graph-69925-7c7b5bdf4d-g6qn6\" (UID: \"ccbd9ee3-f4d3-4d14-970a-257df2ffb815\") " pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6"
May 20 14:03:08.957410 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:08.957371    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccbd9ee3-f4d3-4d14-970a-257df2ffb815-proxy-tls\") pod \"switch-graph-69925-7c7b5bdf4d-g6qn6\" (UID: \"ccbd9ee3-f4d3-4d14-970a-257df2ffb815\") " pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6"
May 20 14:03:08.960117 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:08.960079    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccbd9ee3-f4d3-4d14-970a-257df2ffb815-proxy-tls\") pod \"switch-graph-69925-7c7b5bdf4d-g6qn6\" (UID: \"ccbd9ee3-f4d3-4d14-970a-257df2ffb815\") " pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6"
May 20 14:03:09.165516 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:09.165481    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6"
May 20 14:03:09.305798 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:09.305761    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6"]
May 20 14:03:09.306171 ip-10-0-136-75 kubenswrapper[2570]: W0520 14:03:09.306016    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbd9ee3_f4d3_4d14_970a_257df2ffb815.slice/crio-304a64c18d5219cf9bfede5fabcdca40d0e6c4671a94fc1c96b60274feaaf685 WatchSource:0}: Error finding container 304a64c18d5219cf9bfede5fabcdca40d0e6c4671a94fc1c96b60274feaaf685: Status 404 returned error can't find the container with id 304a64c18d5219cf9bfede5fabcdca40d0e6c4671a94fc1c96b60274feaaf685
May 20 14:03:09.930086 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:09.930039    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6" event={"ID":"ccbd9ee3-f4d3-4d14-970a-257df2ffb815","Type":"ContainerStarted","Data":"304a64c18d5219cf9bfede5fabcdca40d0e6c4671a94fc1c96b60274feaaf685"}
May 20 14:03:11.938522 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:11.938484    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6" event={"ID":"ccbd9ee3-f4d3-4d14-970a-257df2ffb815","Type":"ContainerStarted","Data":"12fe21cb141ab59c2fc841f5401008c9b58f376103b0298ab922b4f338916888"}
May 20 14:03:11.938954 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:11.938592    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6"
May 20 14:03:11.966012 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:11.965965    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6" podStartSLOduration=1.764501087 podStartE2EDuration="3.965952042s" podCreationTimestamp="2026-05-20 14:03:08 +0000 UTC" firstStartedPulling="2026-05-20 14:03:09.30813346 +0000 UTC m=+756.545750086" lastFinishedPulling="2026-05-20 14:03:11.509584407 +0000 UTC m=+758.747201041" observedRunningTime="2026-05-20 14:03:11.963649368 +0000 UTC m=+759.201266011" watchObservedRunningTime="2026-05-20 14:03:11.965952042 +0000 UTC m=+759.203568691"
May 20 14:03:17.948153 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:17.948121    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6"
May 20 14:03:18.320545 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:18.320505    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6"]
May 20 14:03:18.320857 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:18.320784    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6" podUID="ccbd9ee3-f4d3-4d14-970a-257df2ffb815" containerName="switch-graph-69925" containerID="cri-o://12fe21cb141ab59c2fc841f5401008c9b58f376103b0298ab922b4f338916888" gracePeriod=30
May 20 14:03:18.631312 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:18.631177    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"]
May 20 14:03:18.631694 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:18.631649    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4" podUID="3b977c2d-8ee4-4db5-bb33-e828b9e1db25" containerName="kserve-container" containerID="cri-o://49a1592c07cdda4b3740a8e11157408e68813a3caed5ea4e26e09472c4ece120" gracePeriod=30
May 20 14:03:18.631852 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:18.631695    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4" podUID="3b977c2d-8ee4-4db5-bb33-e828b9e1db25" containerName="kube-rbac-proxy" containerID="cri-o://293c1d52e50fa022f9006747b2f5e2391f93c4ceb8e1a3f76ddc6042660ac557" gracePeriod=30
May 20 14:03:18.662427 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:18.662389    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4" podUID="3b977c2d-8ee4-4db5-bb33-e828b9e1db25" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
May 20 14:03:18.688453 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:18.688416    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"]
May 20 14:03:18.690934 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:18.690917    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"
May 20 14:03:18.695193 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:18.695172    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-83116-predictor-serving-cert\""
May 20 14:03:18.695710 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:18.695690    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-83116-kube-rbac-proxy-sar-config\""
May 20 14:03:18.703058 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:18.703033    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"]
May 20 14:03:18.742787 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:18.742764    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghgsg\" (UniqueName: \"kubernetes.io/projected/c5909edf-61bd-432d-bd91-b5e6418f4e8d-kube-api-access-ghgsg\") pod \"error-404-isvc-83116-predictor-849469f6df-2lzrf\" (UID: \"c5909edf-61bd-432d-bd91-b5e6418f4e8d\") " pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"
May 20 14:03:18.742871 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:18.742823    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-83116-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5909edf-61bd-432d-bd91-b5e6418f4e8d-error-404-isvc-83116-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-83116-predictor-849469f6df-2lzrf\" (UID: \"c5909edf-61bd-432d-bd91-b5e6418f4e8d\") " pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"
May 20 14:03:18.742871 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:18.742863    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5909edf-61bd-432d-bd91-b5e6418f4e8d-proxy-tls\") pod \"error-404-isvc-83116-predictor-849469f6df-2lzrf\" (UID: \"c5909edf-61bd-432d-bd91-b5e6418f4e8d\") " pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"
May 20 14:03:18.843919 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:18.843881    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-83116-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5909edf-61bd-432d-bd91-b5e6418f4e8d-error-404-isvc-83116-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-83116-predictor-849469f6df-2lzrf\" (UID: \"c5909edf-61bd-432d-bd91-b5e6418f4e8d\") " pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"
May 20 14:03:18.844103 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:18.843932    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5909edf-61bd-432d-bd91-b5e6418f4e8d-proxy-tls\") pod \"error-404-isvc-83116-predictor-849469f6df-2lzrf\" (UID: \"c5909edf-61bd-432d-bd91-b5e6418f4e8d\") " pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"
May 20 14:03:18.844103 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:18.843976    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ghgsg\" (UniqueName: \"kubernetes.io/projected/c5909edf-61bd-432d-bd91-b5e6418f4e8d-kube-api-access-ghgsg\") pod \"error-404-isvc-83116-predictor-849469f6df-2lzrf\" (UID: \"c5909edf-61bd-432d-bd91-b5e6418f4e8d\") " pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"
May 20 14:03:18.844750 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:18.844692    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-83116-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5909edf-61bd-432d-bd91-b5e6418f4e8d-error-404-isvc-83116-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-83116-predictor-849469f6df-2lzrf\" (UID: \"c5909edf-61bd-432d-bd91-b5e6418f4e8d\") " pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"
May 20 14:03:18.846596 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:18.846575    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5909edf-61bd-432d-bd91-b5e6418f4e8d-proxy-tls\") pod \"error-404-isvc-83116-predictor-849469f6df-2lzrf\" (UID: \"c5909edf-61bd-432d-bd91-b5e6418f4e8d\") " pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"
May 20 14:03:18.854868 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:18.854835    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghgsg\" (UniqueName: \"kubernetes.io/projected/c5909edf-61bd-432d-bd91-b5e6418f4e8d-kube-api-access-ghgsg\") pod \"error-404-isvc-83116-predictor-849469f6df-2lzrf\" (UID: \"c5909edf-61bd-432d-bd91-b5e6418f4e8d\") " pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"
May 20 14:03:18.962372 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:18.962288    2570 generic.go:358] "Generic (PLEG): container finished" podID="3b977c2d-8ee4-4db5-bb33-e828b9e1db25" containerID="293c1d52e50fa022f9006747b2f5e2391f93c4ceb8e1a3f76ddc6042660ac557" exitCode=2
May 20 14:03:18.962372 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:18.962348    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4" event={"ID":"3b977c2d-8ee4-4db5-bb33-e828b9e1db25","Type":"ContainerDied","Data":"293c1d52e50fa022f9006747b2f5e2391f93c4ceb8e1a3f76ddc6042660ac557"}
May 20 14:03:19.002299 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:19.002267    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"
May 20 14:03:19.149135 ip-10-0-136-75 kubenswrapper[2570]: W0520 14:03:19.149102    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5909edf_61bd_432d_bd91_b5e6418f4e8d.slice/crio-ce01d927dbc8cb6369c8aeacd5c377c61ee45fc005b1357e100b61037963b801 WatchSource:0}: Error finding container ce01d927dbc8cb6369c8aeacd5c377c61ee45fc005b1357e100b61037963b801: Status 404 returned error can't find the container with id ce01d927dbc8cb6369c8aeacd5c377c61ee45fc005b1357e100b61037963b801
May 20 14:03:19.149278 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:19.149249    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"]
May 20 14:03:19.966954 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:19.966920    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf" event={"ID":"c5909edf-61bd-432d-bd91-b5e6418f4e8d","Type":"ContainerStarted","Data":"c4ee7872cfc64cfcbad53a71b4e9550cfdb5d77e2446af0462b6dd2cb833908b"}
May 20 14:03:19.966954 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:19.966956    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf" event={"ID":"c5909edf-61bd-432d-bd91-b5e6418f4e8d","Type":"ContainerStarted","Data":"fc038f1f6c9d1775cc24c314478cd23910f5d903cae52ea5ef213fd5709b48ab"}
May 20 14:03:19.967389 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:19.966967    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf" event={"ID":"c5909edf-61bd-432d-bd91-b5e6418f4e8d","Type":"ContainerStarted","Data":"ce01d927dbc8cb6369c8aeacd5c377c61ee45fc005b1357e100b61037963b801"}
May 20 14:03:19.967389 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:19.967054    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"
May 20 14:03:19.991421 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:19.991361    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf" podStartSLOduration=1.99133562 podStartE2EDuration="1.99133562s" podCreationTimestamp="2026-05-20 14:03:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:03:19.990713141 +0000 UTC m=+767.228329789" watchObservedRunningTime="2026-05-20 14:03:19.99133562 +0000 UTC m=+767.228952267"
May 20 14:03:20.972345 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:20.972311    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"
May 20 14:03:20.973424 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:20.973390    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf" podUID="c5909edf-61bd-432d-bd91-b5e6418f4e8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 20 14:03:21.978054 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:21.978015    2570 generic.go:358] "Generic (PLEG): container finished" podID="3b977c2d-8ee4-4db5-bb33-e828b9e1db25" containerID="49a1592c07cdda4b3740a8e11157408e68813a3caed5ea4e26e09472c4ece120" exitCode=0
May 20 14:03:21.978512 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:21.978084    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4" event={"ID":"3b977c2d-8ee4-4db5-bb33-e828b9e1db25","Type":"ContainerDied","Data":"49a1592c07cdda4b3740a8e11157408e68813a3caed5ea4e26e09472c4ece120"}
May 20 14:03:21.978651 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:21.978627    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf" podUID="c5909edf-61bd-432d-bd91-b5e6418f4e8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 20 14:03:22.090749 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:22.090722    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"
May 20 14:03:22.170817 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:22.170727    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xvw7\" (UniqueName: \"kubernetes.io/projected/3b977c2d-8ee4-4db5-bb33-e828b9e1db25-kube-api-access-8xvw7\") pod \"3b977c2d-8ee4-4db5-bb33-e828b9e1db25\" (UID: \"3b977c2d-8ee4-4db5-bb33-e828b9e1db25\") "
May 20 14:03:22.170817 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:22.170775    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-69925-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b977c2d-8ee4-4db5-bb33-e828b9e1db25-error-404-isvc-69925-kube-rbac-proxy-sar-config\") pod \"3b977c2d-8ee4-4db5-bb33-e828b9e1db25\" (UID: \"3b977c2d-8ee4-4db5-bb33-e828b9e1db25\") "
May 20 14:03:22.171047 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:22.170832    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b977c2d-8ee4-4db5-bb33-e828b9e1db25-proxy-tls\") pod \"3b977c2d-8ee4-4db5-bb33-e828b9e1db25\" (UID: \"3b977c2d-8ee4-4db5-bb33-e828b9e1db25\") "
May 20 14:03:22.171160 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:22.171134    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b977c2d-8ee4-4db5-bb33-e828b9e1db25-error-404-isvc-69925-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-69925-kube-rbac-proxy-sar-config") pod "3b977c2d-8ee4-4db5-bb33-e828b9e1db25" (UID: "3b977c2d-8ee4-4db5-bb33-e828b9e1db25"). InnerVolumeSpecName "error-404-isvc-69925-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:03:22.173207 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:22.173165    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b977c2d-8ee4-4db5-bb33-e828b9e1db25-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3b977c2d-8ee4-4db5-bb33-e828b9e1db25" (UID: "3b977c2d-8ee4-4db5-bb33-e828b9e1db25"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:03:22.173307 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:22.173172    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b977c2d-8ee4-4db5-bb33-e828b9e1db25-kube-api-access-8xvw7" (OuterVolumeSpecName: "kube-api-access-8xvw7") pod "3b977c2d-8ee4-4db5-bb33-e828b9e1db25" (UID: "3b977c2d-8ee4-4db5-bb33-e828b9e1db25"). InnerVolumeSpecName "kube-api-access-8xvw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:03:22.271690 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:22.271650    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xvw7\" (UniqueName: \"kubernetes.io/projected/3b977c2d-8ee4-4db5-bb33-e828b9e1db25-kube-api-access-8xvw7\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:03:22.271690 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:22.271689    2570 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-69925-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b977c2d-8ee4-4db5-bb33-e828b9e1db25-error-404-isvc-69925-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:03:22.271897 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:22.271703    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b977c2d-8ee4-4db5-bb33-e828b9e1db25-proxy-tls\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:03:22.946765 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:22.946723    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6" podUID="ccbd9ee3-f4d3-4d14-970a-257df2ffb815" containerName="switch-graph-69925" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:03:22.982836 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:22.982795    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4" event={"ID":"3b977c2d-8ee4-4db5-bb33-e828b9e1db25","Type":"ContainerDied","Data":"e93f17c79288a780ce72e8deaa838ff6df2ebf06735faa1448b46aa25aa0f13f"}
May 20 14:03:22.982836 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:22.982823    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"
May 20 14:03:22.982836 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:22.982844    2570 scope.go:117] "RemoveContainer" containerID="293c1d52e50fa022f9006747b2f5e2391f93c4ceb8e1a3f76ddc6042660ac557"
May 20 14:03:22.991664 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:22.991646    2570 scope.go:117] "RemoveContainer" containerID="49a1592c07cdda4b3740a8e11157408e68813a3caed5ea4e26e09472c4ece120"
May 20 14:03:23.009047 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:23.009013    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"]
May 20 14:03:23.014327 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:23.014298    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-69925-predictor-5d57446965-8zjs4"]
May 20 14:03:23.364327 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:23.364290    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b977c2d-8ee4-4db5-bb33-e828b9e1db25" path="/var/lib/kubelet/pods/3b977c2d-8ee4-4db5-bb33-e828b9e1db25/volumes"
May 20 14:03:26.983421 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:26.983391    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"
May 20 14:03:26.983850 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:26.983818    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf" podUID="c5909edf-61bd-432d-bd91-b5e6418f4e8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 20 14:03:27.945955 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:27.945911    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6" podUID="ccbd9ee3-f4d3-4d14-970a-257df2ffb815" containerName="switch-graph-69925" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:03:32.946652 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:32.946592    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6" podUID="ccbd9ee3-f4d3-4d14-970a-257df2ffb815" containerName="switch-graph-69925" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:03:32.947009 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:32.946741    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6"
May 20 14:03:36.984131 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:36.984087    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf" podUID="c5909edf-61bd-432d-bd91-b5e6418f4e8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 20 14:03:37.946554 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:37.946509    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6" podUID="ccbd9ee3-f4d3-4d14-970a-257df2ffb815" containerName="switch-graph-69925" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:03:38.220925 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:38.220838    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-d669cd755-txm6r"]
May 20 14:03:38.221284 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:38.221197    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b977c2d-8ee4-4db5-bb33-e828b9e1db25" containerName="kube-rbac-proxy"
May 20 14:03:38.221284 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:38.221209    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b977c2d-8ee4-4db5-bb33-e828b9e1db25" containerName="kube-rbac-proxy"
May 20 14:03:38.221284 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:38.221218    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b977c2d-8ee4-4db5-bb33-e828b9e1db25" containerName="kserve-container"
May 20 14:03:38.221284 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:38.221224    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b977c2d-8ee4-4db5-bb33-e828b9e1db25" containerName="kserve-container"
May 20 14:03:38.221402 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:38.221289    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="3b977c2d-8ee4-4db5-bb33-e828b9e1db25" containerName="kube-rbac-proxy"
May 20 14:03:38.221402 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:38.221297    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="3b977c2d-8ee4-4db5-bb33-e828b9e1db25" containerName="kserve-container"
May 20 14:03:38.224456 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:38.224434    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r"
May 20 14:03:38.226969 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:38.226944    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
May 20 14:03:38.227192 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:38.227176    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
May 20 14:03:38.236131 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:38.236103    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-d669cd755-txm6r"]
May 20 14:03:38.311825 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:38.311782    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/324d13b5-bc33-4b21-a680-3ff3788677a1-openshift-service-ca-bundle\") pod \"model-chainer-d669cd755-txm6r\" (UID: \"324d13b5-bc33-4b21-a680-3ff3788677a1\") " pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r"
May 20 14:03:38.312026 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:38.311908    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/324d13b5-bc33-4b21-a680-3ff3788677a1-proxy-tls\") pod \"model-chainer-d669cd755-txm6r\" (UID: \"324d13b5-bc33-4b21-a680-3ff3788677a1\") " pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r"
May 20 14:03:38.412411 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:38.412370    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/324d13b5-bc33-4b21-a680-3ff3788677a1-proxy-tls\") pod \"model-chainer-d669cd755-txm6r\" (UID: \"324d13b5-bc33-4b21-a680-3ff3788677a1\") " pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r"
May 20 14:03:38.412629 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:38.412425    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/324d13b5-bc33-4b21-a680-3ff3788677a1-openshift-service-ca-bundle\") pod \"model-chainer-d669cd755-txm6r\" (UID: \"324d13b5-bc33-4b21-a680-3ff3788677a1\") " pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r"
May 20 14:03:38.412629 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:03:38.412546    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
May 20 14:03:38.412762 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:03:38.412655    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/324d13b5-bc33-4b21-a680-3ff3788677a1-proxy-tls podName:324d13b5-bc33-4b21-a680-3ff3788677a1 nodeName:}" failed. No retries permitted until 2026-05-20 14:03:38.912631416 +0000 UTC m=+786.150248056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/324d13b5-bc33-4b21-a680-3ff3788677a1-proxy-tls") pod "model-chainer-d669cd755-txm6r" (UID: "324d13b5-bc33-4b21-a680-3ff3788677a1") : secret "model-chainer-serving-cert" not found
May 20 14:03:38.413044 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:38.413026    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/324d13b5-bc33-4b21-a680-3ff3788677a1-openshift-service-ca-bundle\") pod \"model-chainer-d669cd755-txm6r\" (UID: \"324d13b5-bc33-4b21-a680-3ff3788677a1\") " pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r"
May 20 14:03:38.916174 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:38.916132    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/324d13b5-bc33-4b21-a680-3ff3788677a1-proxy-tls\") pod \"model-chainer-d669cd755-txm6r\" (UID: \"324d13b5-bc33-4b21-a680-3ff3788677a1\") " pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r"
May 20 14:03:38.918759 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:38.918733    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/324d13b5-bc33-4b21-a680-3ff3788677a1-proxy-tls\") pod \"model-chainer-d669cd755-txm6r\" (UID: \"324d13b5-bc33-4b21-a680-3ff3788677a1\") " pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r"
May 20 14:03:39.135386 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:39.135345    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r"
May 20 14:03:39.274736 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:39.274711    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-d669cd755-txm6r"]
May 20 14:03:40.040895 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:40.040855    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r" event={"ID":"324d13b5-bc33-4b21-a680-3ff3788677a1","Type":"ContainerStarted","Data":"a290ea284bb7a3b97122d6d2caa9cb14cdd2bf1ac251afb704c4cf328d101faf"}
May 20 14:03:40.040895 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:40.040898    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r" event={"ID":"324d13b5-bc33-4b21-a680-3ff3788677a1","Type":"ContainerStarted","Data":"9491b44f3704a92c7349876f455a7b52f15c2e68def3eff646ef0886232ac260"}
May 20 14:03:40.041377 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:40.040999    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r"
May 20 14:03:40.060250 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:40.060197    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r" podStartSLOduration=2.060177079 podStartE2EDuration="2.060177079s" podCreationTimestamp="2026-05-20 14:03:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:03:40.059886702 +0000 UTC m=+787.297503386" watchObservedRunningTime="2026-05-20 14:03:40.060177079 +0000 UTC m=+787.297793727"
May 20 14:03:42.946715 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:42.946670    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6" podUID="ccbd9ee3-f4d3-4d14-970a-257df2ffb815" containerName="switch-graph-69925" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:03:46.050940 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:46.050902    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r"
May 20 14:03:46.984425 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:46.984380    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf" podUID="c5909edf-61bd-432d-bd91-b5e6418f4e8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 20 14:03:47.946982 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:47.946945    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6" podUID="ccbd9ee3-f4d3-4d14-970a-257df2ffb815" containerName="switch-graph-69925" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:03:48.357274 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:03:48.357196    2570 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbd9ee3_f4d3_4d14_970a_257df2ffb815.slice/crio-304a64c18d5219cf9bfede5fabcdca40d0e6c4671a94fc1c96b60274feaaf685\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbd9ee3_f4d3_4d14_970a_257df2ffb815.slice/crio-12fe21cb141ab59c2fc841f5401008c9b58f376103b0298ab922b4f338916888.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbd9ee3_f4d3_4d14_970a_257df2ffb815.slice/crio-conmon-12fe21cb141ab59c2fc841f5401008c9b58f376103b0298ab922b4f338916888.scope\": RecentStats: unable to find data in memory cache]"
May 20 14:03:48.361148 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.361114    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-d669cd755-txm6r"]
May 20 14:03:48.361428 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.361375    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r" podUID="324d13b5-bc33-4b21-a680-3ff3788677a1" containerName="model-chainer" containerID="cri-o://a290ea284bb7a3b97122d6d2caa9cb14cdd2bf1ac251afb704c4cf328d101faf" gracePeriod=30
May 20 14:03:48.363187 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:03:48.363131    2570 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbd9ee3_f4d3_4d14_970a_257df2ffb815.slice/crio-12fe21cb141ab59c2fc841f5401008c9b58f376103b0298ab922b4f338916888.scope\": RecentStats: unable to find data in memory cache]"
May 20 14:03:48.515488 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.515457    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6"
May 20 14:03:48.604518 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.604436    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccbd9ee3-f4d3-4d14-970a-257df2ffb815-openshift-service-ca-bundle\") pod \"ccbd9ee3-f4d3-4d14-970a-257df2ffb815\" (UID: \"ccbd9ee3-f4d3-4d14-970a-257df2ffb815\") "
May 20 14:03:48.604707 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.604537    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccbd9ee3-f4d3-4d14-970a-257df2ffb815-proxy-tls\") pod \"ccbd9ee3-f4d3-4d14-970a-257df2ffb815\" (UID: \"ccbd9ee3-f4d3-4d14-970a-257df2ffb815\") "
May 20 14:03:48.604825 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.604802    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccbd9ee3-f4d3-4d14-970a-257df2ffb815-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ccbd9ee3-f4d3-4d14-970a-257df2ffb815" (UID: "ccbd9ee3-f4d3-4d14-970a-257df2ffb815"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:03:48.606701 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.606679    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccbd9ee3-f4d3-4d14-970a-257df2ffb815-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ccbd9ee3-f4d3-4d14-970a-257df2ffb815" (UID: "ccbd9ee3-f4d3-4d14-970a-257df2ffb815"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:03:48.691559 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.691520    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"]
May 20 14:03:48.692245 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.692219    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccbd9ee3-f4d3-4d14-970a-257df2ffb815" containerName="switch-graph-69925"
May 20 14:03:48.692245 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.692244    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccbd9ee3-f4d3-4d14-970a-257df2ffb815" containerName="switch-graph-69925"
May 20 14:03:48.692414 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.692326    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="ccbd9ee3-f4d3-4d14-970a-257df2ffb815" containerName="switch-graph-69925"
May 20 14:03:48.694662 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.694642    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"
May 20 14:03:48.699243 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.699180    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-cdd1d-predictor-serving-cert\""
May 20 14:03:48.699632 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.699591    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-cdd1d-kube-rbac-proxy-sar-config\""
May 20 14:03:48.706268 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.706247    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccbd9ee3-f4d3-4d14-970a-257df2ffb815-proxy-tls\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:03:48.706362 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.706273    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccbd9ee3-f4d3-4d14-970a-257df2ffb815-openshift-service-ca-bundle\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:03:48.710723 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.710694    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"]
May 20 14:03:48.807576 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.807535    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25bf4864-4618-4763-8476-04e940c2ed2c-proxy-tls\") pod \"error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml\" (UID: \"25bf4864-4618-4763-8476-04e940c2ed2c\") " pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"
May 20 14:03:48.807805 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.807592    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-cdd1d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25bf4864-4618-4763-8476-04e940c2ed2c-error-404-isvc-cdd1d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml\" (UID: \"25bf4864-4618-4763-8476-04e940c2ed2c\") " pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"
May 20 14:03:48.807805 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.807681    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7rv7\" (UniqueName: \"kubernetes.io/projected/25bf4864-4618-4763-8476-04e940c2ed2c-kube-api-access-k7rv7\") pod \"error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml\" (UID: \"25bf4864-4618-4763-8476-04e940c2ed2c\") " pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"
May 20 14:03:48.909137 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.909039    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k7rv7\" (UniqueName: \"kubernetes.io/projected/25bf4864-4618-4763-8476-04e940c2ed2c-kube-api-access-k7rv7\") pod \"error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml\" (UID: \"25bf4864-4618-4763-8476-04e940c2ed2c\") " pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"
May 20 14:03:48.909137 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.909135    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25bf4864-4618-4763-8476-04e940c2ed2c-proxy-tls\") pod \"error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml\" (UID: \"25bf4864-4618-4763-8476-04e940c2ed2c\") " pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"
May 20 14:03:48.909373 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.909187    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-cdd1d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25bf4864-4618-4763-8476-04e940c2ed2c-error-404-isvc-cdd1d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml\" (UID: \"25bf4864-4618-4763-8476-04e940c2ed2c\") " pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"
May 20 14:03:48.909373 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:03:48.909280    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-serving-cert: secret "error-404-isvc-cdd1d-predictor-serving-cert" not found
May 20 14:03:48.909373 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:03:48.909343    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25bf4864-4618-4763-8476-04e940c2ed2c-proxy-tls podName:25bf4864-4618-4763-8476-04e940c2ed2c nodeName:}" failed. No retries permitted until 2026-05-20 14:03:49.409325165 +0000 UTC m=+796.646941791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/25bf4864-4618-4763-8476-04e940c2ed2c-proxy-tls") pod "error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml" (UID: "25bf4864-4618-4763-8476-04e940c2ed2c") : secret "error-404-isvc-cdd1d-predictor-serving-cert" not found
May 20 14:03:48.909922 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.909896    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-cdd1d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25bf4864-4618-4763-8476-04e940c2ed2c-error-404-isvc-cdd1d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml\" (UID: \"25bf4864-4618-4763-8476-04e940c2ed2c\") " pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"
May 20 14:03:48.919587 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:48.919559    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7rv7\" (UniqueName: \"kubernetes.io/projected/25bf4864-4618-4763-8476-04e940c2ed2c-kube-api-access-k7rv7\") pod \"error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml\" (UID: \"25bf4864-4618-4763-8476-04e940c2ed2c\") " pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"
May 20 14:03:49.073304 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:49.073262    2570 generic.go:358] "Generic (PLEG): container finished" podID="ccbd9ee3-f4d3-4d14-970a-257df2ffb815" containerID="12fe21cb141ab59c2fc841f5401008c9b58f376103b0298ab922b4f338916888" exitCode=0
May 20 14:03:49.073799 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:49.073336    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6"
May 20 14:03:49.073799 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:49.073350    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6" event={"ID":"ccbd9ee3-f4d3-4d14-970a-257df2ffb815","Type":"ContainerDied","Data":"12fe21cb141ab59c2fc841f5401008c9b58f376103b0298ab922b4f338916888"}
May 20 14:03:49.073799 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:49.073397    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6" event={"ID":"ccbd9ee3-f4d3-4d14-970a-257df2ffb815","Type":"ContainerDied","Data":"304a64c18d5219cf9bfede5fabcdca40d0e6c4671a94fc1c96b60274feaaf685"}
May 20 14:03:49.073799 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:49.073423    2570 scope.go:117] "RemoveContainer" containerID="12fe21cb141ab59c2fc841f5401008c9b58f376103b0298ab922b4f338916888"
May 20 14:03:49.084028 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:49.084010    2570 scope.go:117] "RemoveContainer" containerID="12fe21cb141ab59c2fc841f5401008c9b58f376103b0298ab922b4f338916888"
May 20 14:03:49.084312 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:03:49.084294    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12fe21cb141ab59c2fc841f5401008c9b58f376103b0298ab922b4f338916888\": container with ID starting with 12fe21cb141ab59c2fc841f5401008c9b58f376103b0298ab922b4f338916888 not found: ID does not exist" containerID="12fe21cb141ab59c2fc841f5401008c9b58f376103b0298ab922b4f338916888"
May 20 14:03:49.084361 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:49.084323    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12fe21cb141ab59c2fc841f5401008c9b58f376103b0298ab922b4f338916888"} err="failed to get container status \"12fe21cb141ab59c2fc841f5401008c9b58f376103b0298ab922b4f338916888\": rpc error: code = NotFound desc = could not find container \"12fe21cb141ab59c2fc841f5401008c9b58f376103b0298ab922b4f338916888\": container with ID starting with 12fe21cb141ab59c2fc841f5401008c9b58f376103b0298ab922b4f338916888 not found: ID does not exist"
May 20 14:03:49.138043 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:49.137978    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6"]
May 20 14:03:49.156725 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:49.156688    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-69925-7c7b5bdf4d-g6qn6"]
May 20 14:03:49.365327 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:49.365287    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccbd9ee3-f4d3-4d14-970a-257df2ffb815" path="/var/lib/kubelet/pods/ccbd9ee3-f4d3-4d14-970a-257df2ffb815/volumes"
May 20 14:03:49.413587 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:49.413545    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25bf4864-4618-4763-8476-04e940c2ed2c-proxy-tls\") pod \"error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml\" (UID: \"25bf4864-4618-4763-8476-04e940c2ed2c\") " pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"
May 20 14:03:49.416472 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:49.416444    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25bf4864-4618-4763-8476-04e940c2ed2c-proxy-tls\") pod \"error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml\" (UID: \"25bf4864-4618-4763-8476-04e940c2ed2c\") " pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"
May 20 14:03:49.608732 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:49.608640    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"
May 20 14:03:49.751961 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:49.751931    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"]
May 20 14:03:49.754447 ip-10-0-136-75 kubenswrapper[2570]: W0520 14:03:49.754415    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25bf4864_4618_4763_8476_04e940c2ed2c.slice/crio-b8ac823354e2089353cb2602769dad7e6947a7d5e2f83defb91e2be78465c593 WatchSource:0}: Error finding container b8ac823354e2089353cb2602769dad7e6947a7d5e2f83defb91e2be78465c593: Status 404 returned error can't find the container with id b8ac823354e2089353cb2602769dad7e6947a7d5e2f83defb91e2be78465c593
May 20 14:03:50.081138 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:50.081091    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml" event={"ID":"25bf4864-4618-4763-8476-04e940c2ed2c","Type":"ContainerStarted","Data":"cf1462429bf778cc3dc592b4a129761a24a6a50bf7abb6c6d1527e389c44a420"}
May 20 14:03:50.081138 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:50.081133    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml" event={"ID":"25bf4864-4618-4763-8476-04e940c2ed2c","Type":"ContainerStarted","Data":"409bfbd3ddaf613a398e51a9c346550f6ee6eed7c47c3314729ac6894a3d5fad"}
May 20 14:03:50.081138 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:50.081148    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml" event={"ID":"25bf4864-4618-4763-8476-04e940c2ed2c","Type":"ContainerStarted","Data":"b8ac823354e2089353cb2602769dad7e6947a7d5e2f83defb91e2be78465c593"}
May 20 14:03:50.081683 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:50.081220    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"
May 20 14:03:50.100985 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:50.100931    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml" podStartSLOduration=2.100915763 podStartE2EDuration="2.100915763s" podCreationTimestamp="2026-05-20 14:03:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:03:50.099922251 +0000 UTC m=+797.337538898" watchObservedRunningTime="2026-05-20 14:03:50.100915763 +0000 UTC m=+797.338532410"
May 20 14:03:51.048884 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:51.048846    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r" podUID="324d13b5-bc33-4b21-a680-3ff3788677a1" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:03:51.085214 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:51.085186    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"
May 20 14:03:51.086592 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:51.086561    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml" podUID="25bf4864-4618-4763-8476-04e940c2ed2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 14:03:52.088271 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:52.088229    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml" podUID="25bf4864-4618-4763-8476-04e940c2ed2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 14:03:56.048601 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:56.048562    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r" podUID="324d13b5-bc33-4b21-a680-3ff3788677a1" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:03:56.983902 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:56.983857    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf" podUID="c5909edf-61bd-432d-bd91-b5e6418f4e8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 20 14:03:57.093215 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:57.093184    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"
May 20 14:03:57.093711 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:03:57.093685    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml" podUID="25bf4864-4618-4763-8476-04e940c2ed2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 14:04:01.048718 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:01.048678    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r" podUID="324d13b5-bc33-4b21-a680-3ff3788677a1" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:04:01.049161 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:01.048816    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r"
May 20 14:04:06.049114 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:06.049071    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r" podUID="324d13b5-bc33-4b21-a680-3ff3788677a1" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:04:06.984448 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:06.984418    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"
May 20 14:04:07.094061 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:07.094021    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml" podUID="25bf4864-4618-4763-8476-04e940c2ed2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 14:04:11.048905 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:11.048865    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r" podUID="324d13b5-bc33-4b21-a680-3ff3788677a1" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:04:16.049010 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:16.048970    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r" podUID="324d13b5-bc33-4b21-a680-3ff3788677a1" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:04:17.094210 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:17.094168    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml" podUID="25bf4864-4618-4763-8476-04e940c2ed2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 14:04:18.600554 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:18.600521    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv"]
May 20 14:04:18.603075 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:18.603057    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv"
May 20 14:04:18.605591 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:18.605559    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-83116-kube-rbac-proxy-sar-config\""
May 20 14:04:18.605746 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:18.605596    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-83116-serving-cert\""
May 20 14:04:18.612703 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:18.612676    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv"]
May 20 14:04:18.663068 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:18.662979    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/096f5b5c-1a00-455e-8523-127f66e27ce5-proxy-tls\") pod \"switch-graph-83116-6fdf4466fc-m5mtv\" (UID: \"096f5b5c-1a00-455e-8523-127f66e27ce5\") " pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv"
May 20 14:04:18.663068 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:18.663029    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/096f5b5c-1a00-455e-8523-127f66e27ce5-openshift-service-ca-bundle\") pod \"switch-graph-83116-6fdf4466fc-m5mtv\" (UID: \"096f5b5c-1a00-455e-8523-127f66e27ce5\") " pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv"
May 20 14:04:18.764220 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:18.764180    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/096f5b5c-1a00-455e-8523-127f66e27ce5-proxy-tls\") pod \"switch-graph-83116-6fdf4466fc-m5mtv\" (UID: \"096f5b5c-1a00-455e-8523-127f66e27ce5\") " pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv"
May 20 14:04:18.764395 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:18.764231    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/096f5b5c-1a00-455e-8523-127f66e27ce5-openshift-service-ca-bundle\") pod \"switch-graph-83116-6fdf4466fc-m5mtv\" (UID: \"096f5b5c-1a00-455e-8523-127f66e27ce5\") " pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv"
May 20 14:04:18.764395 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:04:18.764350    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-83116-serving-cert: secret "switch-graph-83116-serving-cert" not found
May 20 14:04:18.764471 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:04:18.764435    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096f5b5c-1a00-455e-8523-127f66e27ce5-proxy-tls podName:096f5b5c-1a00-455e-8523-127f66e27ce5 nodeName:}" failed. No retries permitted until 2026-05-20 14:04:19.264411549 +0000 UTC m=+826.502028192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/096f5b5c-1a00-455e-8523-127f66e27ce5-proxy-tls") pod "switch-graph-83116-6fdf4466fc-m5mtv" (UID: "096f5b5c-1a00-455e-8523-127f66e27ce5") : secret "switch-graph-83116-serving-cert" not found
May 20 14:04:18.764884 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:18.764867    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/096f5b5c-1a00-455e-8523-127f66e27ce5-openshift-service-ca-bundle\") pod \"switch-graph-83116-6fdf4466fc-m5mtv\" (UID: \"096f5b5c-1a00-455e-8523-127f66e27ce5\") " pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv"
May 20 14:04:19.028465 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:19.028434    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r"
May 20 14:04:19.073131 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:19.073098    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/324d13b5-bc33-4b21-a680-3ff3788677a1-openshift-service-ca-bundle\") pod \"324d13b5-bc33-4b21-a680-3ff3788677a1\" (UID: \"324d13b5-bc33-4b21-a680-3ff3788677a1\") "
May 20 14:04:19.073334 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:19.073179    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/324d13b5-bc33-4b21-a680-3ff3788677a1-proxy-tls\") pod \"324d13b5-bc33-4b21-a680-3ff3788677a1\" (UID: \"324d13b5-bc33-4b21-a680-3ff3788677a1\") "
May 20 14:04:19.073491 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:19.073468    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324d13b5-bc33-4b21-a680-3ff3788677a1-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "324d13b5-bc33-4b21-a680-3ff3788677a1" (UID: "324d13b5-bc33-4b21-a680-3ff3788677a1"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:04:19.075403 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:19.075359    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/324d13b5-bc33-4b21-a680-3ff3788677a1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "324d13b5-bc33-4b21-a680-3ff3788677a1" (UID: "324d13b5-bc33-4b21-a680-3ff3788677a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:04:19.174003 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:19.173960    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/324d13b5-bc33-4b21-a680-3ff3788677a1-openshift-service-ca-bundle\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:04:19.174003 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:19.173989    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/324d13b5-bc33-4b21-a680-3ff3788677a1-proxy-tls\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:04:19.179684 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:19.179645    2570 generic.go:358] "Generic (PLEG): container finished" podID="324d13b5-bc33-4b21-a680-3ff3788677a1" containerID="a290ea284bb7a3b97122d6d2caa9cb14cdd2bf1ac251afb704c4cf328d101faf" exitCode=0
May 20 14:04:19.179891 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:19.179720    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r"
May 20 14:04:19.179891 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:19.179730    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r" event={"ID":"324d13b5-bc33-4b21-a680-3ff3788677a1","Type":"ContainerDied","Data":"a290ea284bb7a3b97122d6d2caa9cb14cdd2bf1ac251afb704c4cf328d101faf"}
May 20 14:04:19.179891 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:19.179764    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-d669cd755-txm6r" event={"ID":"324d13b5-bc33-4b21-a680-3ff3788677a1","Type":"ContainerDied","Data":"9491b44f3704a92c7349876f455a7b52f15c2e68def3eff646ef0886232ac260"}
May 20 14:04:19.179891 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:19.179779    2570 scope.go:117] "RemoveContainer" containerID="a290ea284bb7a3b97122d6d2caa9cb14cdd2bf1ac251afb704c4cf328d101faf"
May 20 14:04:19.188255 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:19.188238    2570 scope.go:117] "RemoveContainer" containerID="a290ea284bb7a3b97122d6d2caa9cb14cdd2bf1ac251afb704c4cf328d101faf"
May 20 14:04:19.188503 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:04:19.188484    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a290ea284bb7a3b97122d6d2caa9cb14cdd2bf1ac251afb704c4cf328d101faf\": container with ID starting with a290ea284bb7a3b97122d6d2caa9cb14cdd2bf1ac251afb704c4cf328d101faf not found: ID does not exist" containerID="a290ea284bb7a3b97122d6d2caa9cb14cdd2bf1ac251afb704c4cf328d101faf"
May 20 14:04:19.188546 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:19.188514    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a290ea284bb7a3b97122d6d2caa9cb14cdd2bf1ac251afb704c4cf328d101faf"} err="failed to get container status \"a290ea284bb7a3b97122d6d2caa9cb14cdd2bf1ac251afb704c4cf328d101faf\": rpc error: code = NotFound desc = could not find container \"a290ea284bb7a3b97122d6d2caa9cb14cdd2bf1ac251afb704c4cf328d101faf\": container with ID starting with a290ea284bb7a3b97122d6d2caa9cb14cdd2bf1ac251afb704c4cf328d101faf not found: ID does not exist"
May 20 14:04:19.203705 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:19.203671    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-d669cd755-txm6r"]
May 20 14:04:19.211039 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:19.211014    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-d669cd755-txm6r"]
May 20 14:04:19.274974 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:19.274945    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/096f5b5c-1a00-455e-8523-127f66e27ce5-proxy-tls\") pod \"switch-graph-83116-6fdf4466fc-m5mtv\" (UID: \"096f5b5c-1a00-455e-8523-127f66e27ce5\") " pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv"
May 20 14:04:19.277642 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:19.277581    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/096f5b5c-1a00-455e-8523-127f66e27ce5-proxy-tls\") pod \"switch-graph-83116-6fdf4466fc-m5mtv\" (UID: \"096f5b5c-1a00-455e-8523-127f66e27ce5\") " pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv"
May 20 14:04:19.364158 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:19.364122    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="324d13b5-bc33-4b21-a680-3ff3788677a1" path="/var/lib/kubelet/pods/324d13b5-bc33-4b21-a680-3ff3788677a1/volumes"
May 20 14:04:19.515112 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:19.515015    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv"
May 20 14:04:19.656055 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:19.656027    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv"]
May 20 14:04:19.658299 ip-10-0-136-75 kubenswrapper[2570]: W0520 14:04:19.658273    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096f5b5c_1a00_455e_8523_127f66e27ce5.slice/crio-cbaf78011d16254104c4dd8e8c7933872948459c0f8711c4a46c71c82f121342 WatchSource:0}: Error finding container cbaf78011d16254104c4dd8e8c7933872948459c0f8711c4a46c71c82f121342: Status 404 returned error can't find the container with id cbaf78011d16254104c4dd8e8c7933872948459c0f8711c4a46c71c82f121342
May 20 14:04:20.185667 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:20.185629    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv" event={"ID":"096f5b5c-1a00-455e-8523-127f66e27ce5","Type":"ContainerStarted","Data":"af1002efddb1953addc505588b852c45b7e4cd8b20883026cb6997809b529e49"}
May 20 14:04:20.185667 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:20.185663    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv" event={"ID":"096f5b5c-1a00-455e-8523-127f66e27ce5","Type":"ContainerStarted","Data":"cbaf78011d16254104c4dd8e8c7933872948459c0f8711c4a46c71c82f121342"}
May 20 14:04:20.185918 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:20.185761    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv"
May 20 14:04:20.204251 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:20.204201    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv" podStartSLOduration=2.204184212 podStartE2EDuration="2.204184212s" podCreationTimestamp="2026-05-20 14:04:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:04:20.203763987 +0000 UTC m=+827.441380637" watchObservedRunningTime="2026-05-20 14:04:20.204184212 +0000 UTC m=+827.441800859"
May 20 14:04:26.195091 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:26.195060    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv"
May 20 14:04:27.094352 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:27.094313    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml" podUID="25bf4864-4618-4763-8476-04e940c2ed2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 14:04:37.094283 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:37.094246    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"
May 20 14:04:48.598693 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:48.598648    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb"]
May 20 14:04:48.599158 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:48.599039    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="324d13b5-bc33-4b21-a680-3ff3788677a1" containerName="model-chainer"
May 20 14:04:48.599158 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:48.599051    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="324d13b5-bc33-4b21-a680-3ff3788677a1" containerName="model-chainer"
May 20 14:04:48.599158 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:48.599124    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="324d13b5-bc33-4b21-a680-3ff3788677a1" containerName="model-chainer"
May 20 14:04:48.601703 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:48.601684    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb"
May 20 14:04:48.606228 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:48.606204    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-cdd1d-kube-rbac-proxy-sar-config\""
May 20 14:04:48.606228 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:48.606218    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-cdd1d-serving-cert\""
May 20 14:04:48.615254 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:48.615230    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb"]
May 20 14:04:48.737427 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:48.737383    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fc80aa-7953-49f6-9d9d-3b3576cd0954-openshift-service-ca-bundle\") pod \"sequence-graph-cdd1d-84988c9c55-pflbb\" (UID: \"f9fc80aa-7953-49f6-9d9d-3b3576cd0954\") " pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb"
May 20 14:04:48.737602 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:48.737500    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9fc80aa-7953-49f6-9d9d-3b3576cd0954-proxy-tls\") pod \"sequence-graph-cdd1d-84988c9c55-pflbb\" (UID: \"f9fc80aa-7953-49f6-9d9d-3b3576cd0954\") " pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb"
May 20 14:04:48.838782 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:48.838735    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fc80aa-7953-49f6-9d9d-3b3576cd0954-openshift-service-ca-bundle\") pod \"sequence-graph-cdd1d-84988c9c55-pflbb\" (UID: \"f9fc80aa-7953-49f6-9d9d-3b3576cd0954\") " pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb"
May 20 14:04:48.838954 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:48.838815    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9fc80aa-7953-49f6-9d9d-3b3576cd0954-proxy-tls\") pod \"sequence-graph-cdd1d-84988c9c55-pflbb\" (UID: \"f9fc80aa-7953-49f6-9d9d-3b3576cd0954\") " pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb"
May 20 14:04:48.839467 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:48.839435    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fc80aa-7953-49f6-9d9d-3b3576cd0954-openshift-service-ca-bundle\") pod \"sequence-graph-cdd1d-84988c9c55-pflbb\" (UID: \"f9fc80aa-7953-49f6-9d9d-3b3576cd0954\") " pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb"
May 20 14:04:48.841390 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:48.841372    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9fc80aa-7953-49f6-9d9d-3b3576cd0954-proxy-tls\") pod \"sequence-graph-cdd1d-84988c9c55-pflbb\" (UID: \"f9fc80aa-7953-49f6-9d9d-3b3576cd0954\") " pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb"
May 20 14:04:48.911480 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:48.911385    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb"
May 20 14:04:49.046579 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:49.046552    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb"]
May 20 14:04:49.048761 ip-10-0-136-75 kubenswrapper[2570]: W0520 14:04:49.048724    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9fc80aa_7953_49f6_9d9d_3b3576cd0954.slice/crio-c501efcdbc8bc35ffa5baf72115f681dad26574eff03d443590db866002d4c41 WatchSource:0}: Error finding container c501efcdbc8bc35ffa5baf72115f681dad26574eff03d443590db866002d4c41: Status 404 returned error can't find the container with id c501efcdbc8bc35ffa5baf72115f681dad26574eff03d443590db866002d4c41
May 20 14:04:49.284059 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:49.284027    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb" event={"ID":"f9fc80aa-7953-49f6-9d9d-3b3576cd0954","Type":"ContainerStarted","Data":"450e8815d4665fde7b8b44a1dbdb3208cbb8476dea99a12940681b945632a0ba"}
May 20 14:04:49.284059 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:49.284063    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb" event={"ID":"f9fc80aa-7953-49f6-9d9d-3b3576cd0954","Type":"ContainerStarted","Data":"c501efcdbc8bc35ffa5baf72115f681dad26574eff03d443590db866002d4c41"}
May 20 14:04:49.284292 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:49.284088    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb"
May 20 14:04:49.312426 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:49.312365    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb" podStartSLOduration=1.312351531 podStartE2EDuration="1.312351531s" podCreationTimestamp="2026-05-20 14:04:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:04:49.311311234 +0000 UTC m=+856.548927883" watchObservedRunningTime="2026-05-20 14:04:49.312351531 +0000 UTC m=+856.549968216"
May 20 14:04:55.293710 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:04:55.293680    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb"
May 20 14:05:33.301284 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:05:33.301251    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 14:05:33.303733 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:05:33.303710    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 14:10:33.324912 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:10:33.324872    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 14:10:33.328152 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:10:33.328123    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 14:12:33.374476 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:33.374442    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv"]
May 20 14:12:33.375824 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:33.374691    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv" podUID="096f5b5c-1a00-455e-8523-127f66e27ce5" containerName="switch-graph-83116" containerID="cri-o://af1002efddb1953addc505588b852c45b7e4cd8b20883026cb6997809b529e49" gracePeriod=30
May 20 14:12:33.827251 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:33.827219    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"]
May 20 14:12:33.827534 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:33.827511    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf" podUID="c5909edf-61bd-432d-bd91-b5e6418f4e8d" containerName="kserve-container" containerID="cri-o://fc038f1f6c9d1775cc24c314478cd23910f5d903cae52ea5ef213fd5709b48ab" gracePeriod=30
May 20 14:12:33.827623 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:33.827586    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf" podUID="c5909edf-61bd-432d-bd91-b5e6418f4e8d" containerName="kube-rbac-proxy" containerID="cri-o://c4ee7872cfc64cfcbad53a71b4e9550cfdb5d77e2446af0462b6dd2cb833908b" gracePeriod=30
May 20 14:12:33.969880 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:33.969846    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"]
May 20 14:12:33.972566 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:33.972542    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"
May 20 14:12:33.993805 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:33.993779    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-15cdb-predictor-serving-cert\""
May 20 14:12:33.997517 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:33.997495    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-15cdb-kube-rbac-proxy-sar-config\""
May 20 14:12:34.095867 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:34.095764    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxtbt\" (UniqueName: \"kubernetes.io/projected/e93a15ed-cdfb-4bb2-b08f-935716638b5a-kube-api-access-mxtbt\") pod \"error-404-isvc-15cdb-predictor-7c646df98f-sjwvp\" (UID: \"e93a15ed-cdfb-4bb2-b08f-935716638b5a\") " pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"
May 20 14:12:34.095867 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:34.095831    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e93a15ed-cdfb-4bb2-b08f-935716638b5a-proxy-tls\") pod \"error-404-isvc-15cdb-predictor-7c646df98f-sjwvp\" (UID: \"e93a15ed-cdfb-4bb2-b08f-935716638b5a\") " pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"
May 20 14:12:34.095867 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:34.095865    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-15cdb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e93a15ed-cdfb-4bb2-b08f-935716638b5a-error-404-isvc-15cdb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-15cdb-predictor-7c646df98f-sjwvp\" (UID: \"e93a15ed-cdfb-4bb2-b08f-935716638b5a\") " pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"
May 20 14:12:34.167705 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:34.167657    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"]
May 20 14:12:34.197173 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:34.197142    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mxtbt\" (UniqueName: \"kubernetes.io/projected/e93a15ed-cdfb-4bb2-b08f-935716638b5a-kube-api-access-mxtbt\") pod \"error-404-isvc-15cdb-predictor-7c646df98f-sjwvp\" (UID: \"e93a15ed-cdfb-4bb2-b08f-935716638b5a\") " pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"
May 20 14:12:34.197353 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:34.197192    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e93a15ed-cdfb-4bb2-b08f-935716638b5a-proxy-tls\") pod \"error-404-isvc-15cdb-predictor-7c646df98f-sjwvp\" (UID: \"e93a15ed-cdfb-4bb2-b08f-935716638b5a\") " pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"
May 20 14:12:34.197353 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:34.197219    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-15cdb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e93a15ed-cdfb-4bb2-b08f-935716638b5a-error-404-isvc-15cdb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-15cdb-predictor-7c646df98f-sjwvp\" (UID: \"e93a15ed-cdfb-4bb2-b08f-935716638b5a\") " pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"
May 20 14:12:34.197911 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:34.197870    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-15cdb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e93a15ed-cdfb-4bb2-b08f-935716638b5a-error-404-isvc-15cdb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-15cdb-predictor-7c646df98f-sjwvp\" (UID: \"e93a15ed-cdfb-4bb2-b08f-935716638b5a\") " pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"
May 20 14:12:34.199860 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:34.199832    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e93a15ed-cdfb-4bb2-b08f-935716638b5a-proxy-tls\") pod \"error-404-isvc-15cdb-predictor-7c646df98f-sjwvp\" (UID: \"e93a15ed-cdfb-4bb2-b08f-935716638b5a\") " pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"
May 20 14:12:34.232849 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:34.232814    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxtbt\" (UniqueName: \"kubernetes.io/projected/e93a15ed-cdfb-4bb2-b08f-935716638b5a-kube-api-access-mxtbt\") pod \"error-404-isvc-15cdb-predictor-7c646df98f-sjwvp\" (UID: \"e93a15ed-cdfb-4bb2-b08f-935716638b5a\") " pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"
May 20 14:12:34.283827 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:34.283792    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"
May 20 14:12:34.443869 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:34.443836    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"]
May 20 14:12:34.445907 ip-10-0-136-75 kubenswrapper[2570]: W0520 14:12:34.445871    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93a15ed_cdfb_4bb2_b08f_935716638b5a.slice/crio-93bcd0cce570e8d02cda7070eeef52e948f0699336a8864e4897bea1e1440273 WatchSource:0}: Error finding container 93bcd0cce570e8d02cda7070eeef52e948f0699336a8864e4897bea1e1440273: Status 404 returned error can't find the container with id 93bcd0cce570e8d02cda7070eeef52e948f0699336a8864e4897bea1e1440273
May 20 14:12:34.447456 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:34.447436    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:12:34.914298 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:34.914257    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp" event={"ID":"e93a15ed-cdfb-4bb2-b08f-935716638b5a","Type":"ContainerStarted","Data":"093bd16fc7a0ad6ef18e933600243923e6034bdd11490ae29906a421ffef9e2d"}
May 20 14:12:34.914298 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:34.914301    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp" event={"ID":"e93a15ed-cdfb-4bb2-b08f-935716638b5a","Type":"ContainerStarted","Data":"0c1c5f2a8b64a287143afaf823441f95a446e1433df4946a176e7413ad775ceb"}
May 20 14:12:34.914559 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:34.914314    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp" event={"ID":"e93a15ed-cdfb-4bb2-b08f-935716638b5a","Type":"ContainerStarted","Data":"93bcd0cce570e8d02cda7070eeef52e948f0699336a8864e4897bea1e1440273"}
May 20 14:12:34.914559 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:34.914458    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"
May 20 14:12:34.914559 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:34.914488    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"
May 20 14:12:34.915689 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:34.915660    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp" podUID="e93a15ed-cdfb-4bb2-b08f-935716638b5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 20 14:12:34.915978 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:34.915960    2570 generic.go:358] "Generic (PLEG): container finished" podID="c5909edf-61bd-432d-bd91-b5e6418f4e8d" containerID="c4ee7872cfc64cfcbad53a71b4e9550cfdb5d77e2446af0462b6dd2cb833908b" exitCode=2
May 20 14:12:34.916064 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:34.916038    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf" event={"ID":"c5909edf-61bd-432d-bd91-b5e6418f4e8d","Type":"ContainerDied","Data":"c4ee7872cfc64cfcbad53a71b4e9550cfdb5d77e2446af0462b6dd2cb833908b"}
May 20 14:12:34.951316 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:34.951256    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp" podStartSLOduration=1.951237031 podStartE2EDuration="1.951237031s" podCreationTimestamp="2026-05-20 14:12:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:12:34.949404878 +0000 UTC m=+1322.187021525" watchObservedRunningTime="2026-05-20 14:12:34.951237031 +0000 UTC m=+1322.188853679"
May 20 14:12:35.920704 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:35.920666    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp" podUID="e93a15ed-cdfb-4bb2-b08f-935716638b5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 20 14:12:36.194047 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:36.193951    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv" podUID="096f5b5c-1a00-455e-8523-127f66e27ce5" containerName="switch-graph-83116" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:12:36.978785 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:36.978740    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf" podUID="c5909edf-61bd-432d-bd91-b5e6418f4e8d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
May 20 14:12:36.984094 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:36.984071    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf" podUID="c5909edf-61bd-432d-bd91-b5e6418f4e8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 20 14:12:37.283252 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:37.283226    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"
May 20 14:12:37.426955 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:37.426920    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-83116-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5909edf-61bd-432d-bd91-b5e6418f4e8d-error-404-isvc-83116-kube-rbac-proxy-sar-config\") pod \"c5909edf-61bd-432d-bd91-b5e6418f4e8d\" (UID: \"c5909edf-61bd-432d-bd91-b5e6418f4e8d\") "
May 20 14:12:37.427135 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:37.426967    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghgsg\" (UniqueName: \"kubernetes.io/projected/c5909edf-61bd-432d-bd91-b5e6418f4e8d-kube-api-access-ghgsg\") pod \"c5909edf-61bd-432d-bd91-b5e6418f4e8d\" (UID: \"c5909edf-61bd-432d-bd91-b5e6418f4e8d\") "
May 20 14:12:37.427135 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:37.427048    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5909edf-61bd-432d-bd91-b5e6418f4e8d-proxy-tls\") pod \"c5909edf-61bd-432d-bd91-b5e6418f4e8d\" (UID: \"c5909edf-61bd-432d-bd91-b5e6418f4e8d\") "
May 20 14:12:37.427364 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:37.427327    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5909edf-61bd-432d-bd91-b5e6418f4e8d-error-404-isvc-83116-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-83116-kube-rbac-proxy-sar-config") pod "c5909edf-61bd-432d-bd91-b5e6418f4e8d" (UID: "c5909edf-61bd-432d-bd91-b5e6418f4e8d"). InnerVolumeSpecName "error-404-isvc-83116-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:12:37.429288 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:37.429252    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5909edf-61bd-432d-bd91-b5e6418f4e8d-kube-api-access-ghgsg" (OuterVolumeSpecName: "kube-api-access-ghgsg") pod "c5909edf-61bd-432d-bd91-b5e6418f4e8d" (UID: "c5909edf-61bd-432d-bd91-b5e6418f4e8d"). InnerVolumeSpecName "kube-api-access-ghgsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:12:37.429288 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:37.429251    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5909edf-61bd-432d-bd91-b5e6418f4e8d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c5909edf-61bd-432d-bd91-b5e6418f4e8d" (UID: "c5909edf-61bd-432d-bd91-b5e6418f4e8d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:12:37.528383 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:37.528358    2570 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-83116-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5909edf-61bd-432d-bd91-b5e6418f4e8d-error-404-isvc-83116-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:12:37.528383 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:37.528382    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghgsg\" (UniqueName: \"kubernetes.io/projected/c5909edf-61bd-432d-bd91-b5e6418f4e8d-kube-api-access-ghgsg\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:12:37.528567 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:37.528393    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5909edf-61bd-432d-bd91-b5e6418f4e8d-proxy-tls\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:12:37.928467 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:37.928377    2570 generic.go:358] "Generic (PLEG): container finished" podID="c5909edf-61bd-432d-bd91-b5e6418f4e8d" containerID="fc038f1f6c9d1775cc24c314478cd23910f5d903cae52ea5ef213fd5709b48ab" exitCode=0
May 20 14:12:37.928467 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:37.928449    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"
May 20 14:12:37.928467 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:37.928455    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf" event={"ID":"c5909edf-61bd-432d-bd91-b5e6418f4e8d","Type":"ContainerDied","Data":"fc038f1f6c9d1775cc24c314478cd23910f5d903cae52ea5ef213fd5709b48ab"}
May 20 14:12:37.928784 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:37.928493    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf" event={"ID":"c5909edf-61bd-432d-bd91-b5e6418f4e8d","Type":"ContainerDied","Data":"ce01d927dbc8cb6369c8aeacd5c377c61ee45fc005b1357e100b61037963b801"}
May 20 14:12:37.928784 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:37.928513    2570 scope.go:117] "RemoveContainer" containerID="c4ee7872cfc64cfcbad53a71b4e9550cfdb5d77e2446af0462b6dd2cb833908b"
May 20 14:12:37.937677 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:37.937659    2570 scope.go:117] "RemoveContainer" containerID="fc038f1f6c9d1775cc24c314478cd23910f5d903cae52ea5ef213fd5709b48ab"
May 20 14:12:37.945115 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:37.945098    2570 scope.go:117] "RemoveContainer" containerID="c4ee7872cfc64cfcbad53a71b4e9550cfdb5d77e2446af0462b6dd2cb833908b"
May 20 14:12:37.945343 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:12:37.945323    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4ee7872cfc64cfcbad53a71b4e9550cfdb5d77e2446af0462b6dd2cb833908b\": container with ID starting with c4ee7872cfc64cfcbad53a71b4e9550cfdb5d77e2446af0462b6dd2cb833908b not found: ID does not exist" containerID="c4ee7872cfc64cfcbad53a71b4e9550cfdb5d77e2446af0462b6dd2cb833908b"
May 20 14:12:37.945412 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:37.945358    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4ee7872cfc64cfcbad53a71b4e9550cfdb5d77e2446af0462b6dd2cb833908b"} err="failed to get container status \"c4ee7872cfc64cfcbad53a71b4e9550cfdb5d77e2446af0462b6dd2cb833908b\": rpc error: code = NotFound desc = could not find container \"c4ee7872cfc64cfcbad53a71b4e9550cfdb5d77e2446af0462b6dd2cb833908b\": container with ID starting with c4ee7872cfc64cfcbad53a71b4e9550cfdb5d77e2446af0462b6dd2cb833908b not found: ID does not exist"
May 20 14:12:37.945412 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:37.945386    2570 scope.go:117] "RemoveContainer" containerID="fc038f1f6c9d1775cc24c314478cd23910f5d903cae52ea5ef213fd5709b48ab"
May 20 14:12:37.945685 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:12:37.945664    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc038f1f6c9d1775cc24c314478cd23910f5d903cae52ea5ef213fd5709b48ab\": container with ID starting with fc038f1f6c9d1775cc24c314478cd23910f5d903cae52ea5ef213fd5709b48ab not found: ID does not exist" containerID="fc038f1f6c9d1775cc24c314478cd23910f5d903cae52ea5ef213fd5709b48ab"
May 20 14:12:37.945732 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:37.945697    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc038f1f6c9d1775cc24c314478cd23910f5d903cae52ea5ef213fd5709b48ab"} err="failed to get container status \"fc038f1f6c9d1775cc24c314478cd23910f5d903cae52ea5ef213fd5709b48ab\": rpc error: code = NotFound desc = could not find container \"fc038f1f6c9d1775cc24c314478cd23910f5d903cae52ea5ef213fd5709b48ab\": container with ID starting with fc038f1f6c9d1775cc24c314478cd23910f5d903cae52ea5ef213fd5709b48ab not found: ID does not exist"
May 20 14:12:37.954485 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:37.954459    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"]
May 20 14:12:37.959664 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:37.959637    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-83116-predictor-849469f6df-2lzrf"]
May 20 14:12:39.365115 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:39.365081    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5909edf-61bd-432d-bd91-b5e6418f4e8d" path="/var/lib/kubelet/pods/c5909edf-61bd-432d-bd91-b5e6418f4e8d/volumes"
May 20 14:12:40.926547 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:40.926516    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"
May 20 14:12:40.927073 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:40.927047    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp" podUID="e93a15ed-cdfb-4bb2-b08f-935716638b5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 20 14:12:41.193886 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:41.193796    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv" podUID="096f5b5c-1a00-455e-8523-127f66e27ce5" containerName="switch-graph-83116" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:12:46.193115 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:46.193067    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv" podUID="096f5b5c-1a00-455e-8523-127f66e27ce5" containerName="switch-graph-83116" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:12:46.193499 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:46.193183    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv"
May 20 14:12:50.927125 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:50.927035    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp" podUID="e93a15ed-cdfb-4bb2-b08f-935716638b5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 20 14:12:51.194341 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:51.194250    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv" podUID="096f5b5c-1a00-455e-8523-127f66e27ce5" containerName="switch-graph-83116" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:12:56.193244 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:12:56.193203    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv" podUID="096f5b5c-1a00-455e-8523-127f66e27ce5" containerName="switch-graph-83116" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:13:00.927686 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:00.927648    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp" podUID="e93a15ed-cdfb-4bb2-b08f-935716638b5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 20 14:13:01.193899 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:01.193807    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv" podUID="096f5b5c-1a00-455e-8523-127f66e27ce5" containerName="switch-graph-83116" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:13:03.250901 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.250867    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb"]
May 20 14:13:03.251306 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.251125    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb" podUID="f9fc80aa-7953-49f6-9d9d-3b3576cd0954" containerName="sequence-graph-cdd1d" containerID="cri-o://450e8815d4665fde7b8b44a1dbdb3208cbb8476dea99a12940681b945632a0ba" gracePeriod=30
May 20 14:13:03.534559 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.534322    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv"
May 20 14:13:03.564655 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.564597    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/096f5b5c-1a00-455e-8523-127f66e27ce5-openshift-service-ca-bundle\") pod \"096f5b5c-1a00-455e-8523-127f66e27ce5\" (UID: \"096f5b5c-1a00-455e-8523-127f66e27ce5\") "
May 20 14:13:03.564834 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.564801    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/096f5b5c-1a00-455e-8523-127f66e27ce5-proxy-tls\") pod \"096f5b5c-1a00-455e-8523-127f66e27ce5\" (UID: \"096f5b5c-1a00-455e-8523-127f66e27ce5\") "
May 20 14:13:03.565010 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.564984    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096f5b5c-1a00-455e-8523-127f66e27ce5-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "096f5b5c-1a00-455e-8523-127f66e27ce5" (UID: "096f5b5c-1a00-455e-8523-127f66e27ce5"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:13:03.565118 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.565101    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/096f5b5c-1a00-455e-8523-127f66e27ce5-openshift-service-ca-bundle\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:13:03.567210 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.567184    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/096f5b5c-1a00-455e-8523-127f66e27ce5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "096f5b5c-1a00-455e-8523-127f66e27ce5" (UID: "096f5b5c-1a00-455e-8523-127f66e27ce5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:13:03.619166 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.619120    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"]
May 20 14:13:03.619474 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.619416    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml" podUID="25bf4864-4618-4763-8476-04e940c2ed2c" containerName="kserve-container" containerID="cri-o://409bfbd3ddaf613a398e51a9c346550f6ee6eed7c47c3314729ac6894a3d5fad" gracePeriod=30
May 20 14:13:03.619597 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.619450    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml" podUID="25bf4864-4618-4763-8476-04e940c2ed2c" containerName="kube-rbac-proxy" containerID="cri-o://cf1462429bf778cc3dc592b4a129761a24a6a50bf7abb6c6d1527e389c44a420" gracePeriod=30
May 20 14:13:03.666545 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.666512    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/096f5b5c-1a00-455e-8523-127f66e27ce5-proxy-tls\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:13:03.694003 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.693962    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"]
May 20 14:13:03.694515 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.694496    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5909edf-61bd-432d-bd91-b5e6418f4e8d" containerName="kserve-container"
May 20 14:13:03.694572 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.694518    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5909edf-61bd-432d-bd91-b5e6418f4e8d" containerName="kserve-container"
May 20 14:13:03.694572 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.694527    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5909edf-61bd-432d-bd91-b5e6418f4e8d" containerName="kube-rbac-proxy"
May 20 14:13:03.694572 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.694533    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5909edf-61bd-432d-bd91-b5e6418f4e8d" containerName="kube-rbac-proxy"
May 20 14:13:03.694572 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.694547    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="096f5b5c-1a00-455e-8523-127f66e27ce5" containerName="switch-graph-83116"
May 20 14:13:03.694572 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.694553    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="096f5b5c-1a00-455e-8523-127f66e27ce5" containerName="switch-graph-83116"
May 20 14:13:03.694756 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.694649    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="c5909edf-61bd-432d-bd91-b5e6418f4e8d" containerName="kserve-container"
May 20 14:13:03.694756 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.694660    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="c5909edf-61bd-432d-bd91-b5e6418f4e8d" containerName="kube-rbac-proxy"
May 20 14:13:03.694756 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.694669    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="096f5b5c-1a00-455e-8523-127f66e27ce5" containerName="switch-graph-83116"
May 20 14:13:03.696871 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.696852    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"
May 20 14:13:03.701707 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.701686    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-14a90-predictor-serving-cert\""
May 20 14:13:03.702249 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.702229    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-14a90-kube-rbac-proxy-sar-config\""
May 20 14:13:03.716524 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.716496    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"]
May 20 14:13:03.767273 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.767194    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqwp7\" (UniqueName: \"kubernetes.io/projected/677266f2-8375-4dcf-a101-0160c6f8f5b4-kube-api-access-lqwp7\") pod \"error-404-isvc-14a90-predictor-547774646f-lrxtg\" (UID: \"677266f2-8375-4dcf-a101-0160c6f8f5b4\") " pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"
May 20 14:13:03.767273 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.767250    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-14a90-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/677266f2-8375-4dcf-a101-0160c6f8f5b4-error-404-isvc-14a90-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-14a90-predictor-547774646f-lrxtg\" (UID: \"677266f2-8375-4dcf-a101-0160c6f8f5b4\") " pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"
May 20 14:13:03.767464 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.767389    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/677266f2-8375-4dcf-a101-0160c6f8f5b4-proxy-tls\") pod \"error-404-isvc-14a90-predictor-547774646f-lrxtg\" (UID: \"677266f2-8375-4dcf-a101-0160c6f8f5b4\") " pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"
May 20 14:13:03.868517 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.868482    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/677266f2-8375-4dcf-a101-0160c6f8f5b4-proxy-tls\") pod \"error-404-isvc-14a90-predictor-547774646f-lrxtg\" (UID: \"677266f2-8375-4dcf-a101-0160c6f8f5b4\") " pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"
May 20 14:13:03.868733 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.868540    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lqwp7\" (UniqueName: \"kubernetes.io/projected/677266f2-8375-4dcf-a101-0160c6f8f5b4-kube-api-access-lqwp7\") pod \"error-404-isvc-14a90-predictor-547774646f-lrxtg\" (UID: \"677266f2-8375-4dcf-a101-0160c6f8f5b4\") " pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"
May 20 14:13:03.868733 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.868585    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-14a90-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/677266f2-8375-4dcf-a101-0160c6f8f5b4-error-404-isvc-14a90-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-14a90-predictor-547774646f-lrxtg\" (UID: \"677266f2-8375-4dcf-a101-0160c6f8f5b4\") " pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"
May 20 14:13:03.868733 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:13:03.868672    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-14a90-predictor-serving-cert: secret "error-404-isvc-14a90-predictor-serving-cert" not found
May 20 14:13:03.868882 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:13:03.868749    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/677266f2-8375-4dcf-a101-0160c6f8f5b4-proxy-tls podName:677266f2-8375-4dcf-a101-0160c6f8f5b4 nodeName:}" failed. No retries permitted until 2026-05-20 14:13:04.368727198 +0000 UTC m=+1351.606343841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/677266f2-8375-4dcf-a101-0160c6f8f5b4-proxy-tls") pod "error-404-isvc-14a90-predictor-547774646f-lrxtg" (UID: "677266f2-8375-4dcf-a101-0160c6f8f5b4") : secret "error-404-isvc-14a90-predictor-serving-cert" not found
May 20 14:13:03.869278 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.869261    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-14a90-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/677266f2-8375-4dcf-a101-0160c6f8f5b4-error-404-isvc-14a90-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-14a90-predictor-547774646f-lrxtg\" (UID: \"677266f2-8375-4dcf-a101-0160c6f8f5b4\") " pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"
May 20 14:13:03.880523 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:03.880493    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqwp7\" (UniqueName: \"kubernetes.io/projected/677266f2-8375-4dcf-a101-0160c6f8f5b4-kube-api-access-lqwp7\") pod \"error-404-isvc-14a90-predictor-547774646f-lrxtg\" (UID: \"677266f2-8375-4dcf-a101-0160c6f8f5b4\") " pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"
May 20 14:13:04.023271 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:04.023237    2570 generic.go:358] "Generic (PLEG): container finished" podID="25bf4864-4618-4763-8476-04e940c2ed2c" containerID="cf1462429bf778cc3dc592b4a129761a24a6a50bf7abb6c6d1527e389c44a420" exitCode=2
May 20 14:13:04.023446 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:04.023307    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml" event={"ID":"25bf4864-4618-4763-8476-04e940c2ed2c","Type":"ContainerDied","Data":"cf1462429bf778cc3dc592b4a129761a24a6a50bf7abb6c6d1527e389c44a420"}
May 20 14:13:04.024383 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:04.024361    2570 generic.go:358] "Generic (PLEG): container finished" podID="096f5b5c-1a00-455e-8523-127f66e27ce5" containerID="af1002efddb1953addc505588b852c45b7e4cd8b20883026cb6997809b529e49" exitCode=0
May 20 14:13:04.024475 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:04.024427    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv"
May 20 14:13:04.024475 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:04.024445    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv" event={"ID":"096f5b5c-1a00-455e-8523-127f66e27ce5","Type":"ContainerDied","Data":"af1002efddb1953addc505588b852c45b7e4cd8b20883026cb6997809b529e49"}
May 20 14:13:04.024555 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:04.024479    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv" event={"ID":"096f5b5c-1a00-455e-8523-127f66e27ce5","Type":"ContainerDied","Data":"cbaf78011d16254104c4dd8e8c7933872948459c0f8711c4a46c71c82f121342"}
May 20 14:13:04.024555 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:04.024496    2570 scope.go:117] "RemoveContainer" containerID="af1002efddb1953addc505588b852c45b7e4cd8b20883026cb6997809b529e49"
May 20 14:13:04.035300 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:04.035200    2570 scope.go:117] "RemoveContainer" containerID="af1002efddb1953addc505588b852c45b7e4cd8b20883026cb6997809b529e49"
May 20 14:13:04.035758 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:13:04.035736    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1002efddb1953addc505588b852c45b7e4cd8b20883026cb6997809b529e49\": container with ID starting with af1002efddb1953addc505588b852c45b7e4cd8b20883026cb6997809b529e49 not found: ID does not exist" containerID="af1002efddb1953addc505588b852c45b7e4cd8b20883026cb6997809b529e49"
May 20 14:13:04.035849 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:04.035771    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1002efddb1953addc505588b852c45b7e4cd8b20883026cb6997809b529e49"} err="failed to get container status \"af1002efddb1953addc505588b852c45b7e4cd8b20883026cb6997809b529e49\": rpc error: code = NotFound desc = could not find container \"af1002efddb1953addc505588b852c45b7e4cd8b20883026cb6997809b529e49\": container with ID starting with af1002efddb1953addc505588b852c45b7e4cd8b20883026cb6997809b529e49 not found: ID does not exist"
May 20 14:13:04.048813 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:04.048778    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv"]
May 20 14:13:04.056794 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:04.056771    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-83116-6fdf4466fc-m5mtv"]
May 20 14:13:04.374564 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:04.374455    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/677266f2-8375-4dcf-a101-0160c6f8f5b4-proxy-tls\") pod \"error-404-isvc-14a90-predictor-547774646f-lrxtg\" (UID: \"677266f2-8375-4dcf-a101-0160c6f8f5b4\") " pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"
May 20 14:13:04.377242 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:04.377211    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/677266f2-8375-4dcf-a101-0160c6f8f5b4-proxy-tls\") pod \"error-404-isvc-14a90-predictor-547774646f-lrxtg\" (UID: \"677266f2-8375-4dcf-a101-0160c6f8f5b4\") " pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"
May 20 14:13:04.607481 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:04.607442    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"
May 20 14:13:04.741968 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:04.741936    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"]
May 20 14:13:04.745804 ip-10-0-136-75 kubenswrapper[2570]: W0520 14:13:04.745774    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677266f2_8375_4dcf_a101_0160c6f8f5b4.slice/crio-f8f87f65dab0a769fb04a5744ca4ee5f51db0300baa97e78227bc6db13dc084b WatchSource:0}: Error finding container f8f87f65dab0a769fb04a5744ca4ee5f51db0300baa97e78227bc6db13dc084b: Status 404 returned error can't find the container with id f8f87f65dab0a769fb04a5744ca4ee5f51db0300baa97e78227bc6db13dc084b
May 20 14:13:05.029349 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:05.029315    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg" event={"ID":"677266f2-8375-4dcf-a101-0160c6f8f5b4","Type":"ContainerStarted","Data":"e1e398541b2fa91372dc5a3236d371cc258ac181839d44162982022181010609"}
May 20 14:13:05.029552 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:05.029356    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg" event={"ID":"677266f2-8375-4dcf-a101-0160c6f8f5b4","Type":"ContainerStarted","Data":"4605bb3d807b86027680bc49e7fc40743c28cd3fdd03302a2e80ab2101794de4"}
May 20 14:13:05.029552 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:05.029372    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg" event={"ID":"677266f2-8375-4dcf-a101-0160c6f8f5b4","Type":"ContainerStarted","Data":"f8f87f65dab0a769fb04a5744ca4ee5f51db0300baa97e78227bc6db13dc084b"}
May 20 14:13:05.029552 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:05.029449    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"
May 20 14:13:05.053531 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:05.053471    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg" podStartSLOduration=2.053454682 podStartE2EDuration="2.053454682s" podCreationTimestamp="2026-05-20 14:13:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:13:05.053397444 +0000 UTC m=+1352.291014093" watchObservedRunningTime="2026-05-20 14:13:05.053454682 +0000 UTC m=+1352.291071330"
May 20 14:13:05.292376 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:05.292283    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb" podUID="f9fc80aa-7953-49f6-9d9d-3b3576cd0954" containerName="sequence-graph-cdd1d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:13:05.365042 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:05.365005    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="096f5b5c-1a00-455e-8523-127f66e27ce5" path="/var/lib/kubelet/pods/096f5b5c-1a00-455e-8523-127f66e27ce5/volumes"
May 20 14:13:06.034328 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:06.034292    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"
May 20 14:13:06.035790 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:06.035758    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg" podUID="677266f2-8375-4dcf-a101-0160c6f8f5b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 14:13:06.980569 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:06.980538    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"
May 20 14:13:07.039296 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:07.039207    2570 generic.go:358] "Generic (PLEG): container finished" podID="25bf4864-4618-4763-8476-04e940c2ed2c" containerID="409bfbd3ddaf613a398e51a9c346550f6ee6eed7c47c3314729ac6894a3d5fad" exitCode=0
May 20 14:13:07.039788 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:07.039291    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml" event={"ID":"25bf4864-4618-4763-8476-04e940c2ed2c","Type":"ContainerDied","Data":"409bfbd3ddaf613a398e51a9c346550f6ee6eed7c47c3314729ac6894a3d5fad"}
May 20 14:13:07.039788 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:07.039303    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"
May 20 14:13:07.039788 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:07.039350    2570 scope.go:117] "RemoveContainer" containerID="cf1462429bf778cc3dc592b4a129761a24a6a50bf7abb6c6d1527e389c44a420"
May 20 14:13:07.039788 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:07.039336    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml" event={"ID":"25bf4864-4618-4763-8476-04e940c2ed2c","Type":"ContainerDied","Data":"b8ac823354e2089353cb2602769dad7e6947a7d5e2f83defb91e2be78465c593"}
May 20 14:13:07.040025 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:07.040004    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg" podUID="677266f2-8375-4dcf-a101-0160c6f8f5b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 14:13:07.048497 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:07.048468    2570 scope.go:117] "RemoveContainer" containerID="409bfbd3ddaf613a398e51a9c346550f6ee6eed7c47c3314729ac6894a3d5fad"
May 20 14:13:07.056299 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:07.056278    2570 scope.go:117] "RemoveContainer" containerID="cf1462429bf778cc3dc592b4a129761a24a6a50bf7abb6c6d1527e389c44a420"
May 20 14:13:07.056553 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:13:07.056533    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf1462429bf778cc3dc592b4a129761a24a6a50bf7abb6c6d1527e389c44a420\": container with ID starting with cf1462429bf778cc3dc592b4a129761a24a6a50bf7abb6c6d1527e389c44a420 not found: ID does not exist" containerID="cf1462429bf778cc3dc592b4a129761a24a6a50bf7abb6c6d1527e389c44a420"
May 20 14:13:07.056603 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:07.056562    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1462429bf778cc3dc592b4a129761a24a6a50bf7abb6c6d1527e389c44a420"} err="failed to get container status \"cf1462429bf778cc3dc592b4a129761a24a6a50bf7abb6c6d1527e389c44a420\": rpc error: code = NotFound desc = could not find container \"cf1462429bf778cc3dc592b4a129761a24a6a50bf7abb6c6d1527e389c44a420\": container with ID starting with cf1462429bf778cc3dc592b4a129761a24a6a50bf7abb6c6d1527e389c44a420 not found: ID does not exist"
May 20 14:13:07.056603 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:07.056581    2570 scope.go:117] "RemoveContainer" containerID="409bfbd3ddaf613a398e51a9c346550f6ee6eed7c47c3314729ac6894a3d5fad"
May 20 14:13:07.056815 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:13:07.056795    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"409bfbd3ddaf613a398e51a9c346550f6ee6eed7c47c3314729ac6894a3d5fad\": container with ID starting with 409bfbd3ddaf613a398e51a9c346550f6ee6eed7c47c3314729ac6894a3d5fad not found: ID does not exist" containerID="409bfbd3ddaf613a398e51a9c346550f6ee6eed7c47c3314729ac6894a3d5fad"
May 20 14:13:07.056858 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:07.056821    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"409bfbd3ddaf613a398e51a9c346550f6ee6eed7c47c3314729ac6894a3d5fad"} err="failed to get container status \"409bfbd3ddaf613a398e51a9c346550f6ee6eed7c47c3314729ac6894a3d5fad\": rpc error: code = NotFound desc = could not find container \"409bfbd3ddaf613a398e51a9c346550f6ee6eed7c47c3314729ac6894a3d5fad\": container with ID starting with 409bfbd3ddaf613a398e51a9c346550f6ee6eed7c47c3314729ac6894a3d5fad not found: ID does not exist"
May 20 14:13:07.100359 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:07.100323    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-cdd1d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25bf4864-4618-4763-8476-04e940c2ed2c-error-404-isvc-cdd1d-kube-rbac-proxy-sar-config\") pod \"25bf4864-4618-4763-8476-04e940c2ed2c\" (UID: \"25bf4864-4618-4763-8476-04e940c2ed2c\") "
May 20 14:13:07.100556 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:07.100382    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25bf4864-4618-4763-8476-04e940c2ed2c-proxy-tls\") pod \"25bf4864-4618-4763-8476-04e940c2ed2c\" (UID: \"25bf4864-4618-4763-8476-04e940c2ed2c\") "
May 20 14:13:07.100556 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:07.100448    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7rv7\" (UniqueName: \"kubernetes.io/projected/25bf4864-4618-4763-8476-04e940c2ed2c-kube-api-access-k7rv7\") pod \"25bf4864-4618-4763-8476-04e940c2ed2c\" (UID: \"25bf4864-4618-4763-8476-04e940c2ed2c\") "
May 20 14:13:07.100839 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:07.100793    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25bf4864-4618-4763-8476-04e940c2ed2c-error-404-isvc-cdd1d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-cdd1d-kube-rbac-proxy-sar-config") pod "25bf4864-4618-4763-8476-04e940c2ed2c" (UID: "25bf4864-4618-4763-8476-04e940c2ed2c"). InnerVolumeSpecName "error-404-isvc-cdd1d-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:13:07.102767 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:07.102742    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25bf4864-4618-4763-8476-04e940c2ed2c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "25bf4864-4618-4763-8476-04e940c2ed2c" (UID: "25bf4864-4618-4763-8476-04e940c2ed2c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:13:07.102856 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:07.102822    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25bf4864-4618-4763-8476-04e940c2ed2c-kube-api-access-k7rv7" (OuterVolumeSpecName: "kube-api-access-k7rv7") pod "25bf4864-4618-4763-8476-04e940c2ed2c" (UID: "25bf4864-4618-4763-8476-04e940c2ed2c"). InnerVolumeSpecName "kube-api-access-k7rv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:13:07.201699 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:07.201656    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7rv7\" (UniqueName: \"kubernetes.io/projected/25bf4864-4618-4763-8476-04e940c2ed2c-kube-api-access-k7rv7\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:13:07.201699 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:07.201696    2570 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-cdd1d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25bf4864-4618-4763-8476-04e940c2ed2c-error-404-isvc-cdd1d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:13:07.201908 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:07.201715    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25bf4864-4618-4763-8476-04e940c2ed2c-proxy-tls\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:13:07.364829 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:07.364799    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"]
May 20 14:13:07.365655 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:07.365634    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cdd1d-predictor-66cb4945ff-2r8ml"]
May 20 14:13:09.364281 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:09.364248    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25bf4864-4618-4763-8476-04e940c2ed2c" path="/var/lib/kubelet/pods/25bf4864-4618-4763-8476-04e940c2ed2c/volumes"
May 20 14:13:10.292601 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:10.292560    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb" podUID="f9fc80aa-7953-49f6-9d9d-3b3576cd0954" containerName="sequence-graph-cdd1d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:13:10.927868 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:10.927826    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp" podUID="e93a15ed-cdfb-4bb2-b08f-935716638b5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 20 14:13:12.044038 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:12.044011    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"
May 20 14:13:12.044495 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:12.044420    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg" podUID="677266f2-8375-4dcf-a101-0160c6f8f5b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 14:13:15.292408 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:15.292361    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb" podUID="f9fc80aa-7953-49f6-9d9d-3b3576cd0954" containerName="sequence-graph-cdd1d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:13:15.292874 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:15.292501    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb"
May 20 14:13:20.292142 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:20.292105    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb" podUID="f9fc80aa-7953-49f6-9d9d-3b3576cd0954" containerName="sequence-graph-cdd1d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:13:20.927782 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:20.927753    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"
May 20 14:13:22.044541 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:22.044502    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg" podUID="677266f2-8375-4dcf-a101-0160c6f8f5b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 14:13:25.291959 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:25.291916    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb" podUID="f9fc80aa-7953-49f6-9d9d-3b3576cd0954" containerName="sequence-graph-cdd1d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:13:30.292168 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:30.292121    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb" podUID="f9fc80aa-7953-49f6-9d9d-3b3576cd0954" containerName="sequence-graph-cdd1d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:13:32.044761 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:32.044722    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg" podUID="677266f2-8375-4dcf-a101-0160c6f8f5b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 14:13:33.453154 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.453129    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb"
May 20 14:13:33.540230 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.540201    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fc80aa-7953-49f6-9d9d-3b3576cd0954-openshift-service-ca-bundle\") pod \"f9fc80aa-7953-49f6-9d9d-3b3576cd0954\" (UID: \"f9fc80aa-7953-49f6-9d9d-3b3576cd0954\") "
May 20 14:13:33.540398 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.540276    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9fc80aa-7953-49f6-9d9d-3b3576cd0954-proxy-tls\") pod \"f9fc80aa-7953-49f6-9d9d-3b3576cd0954\" (UID: \"f9fc80aa-7953-49f6-9d9d-3b3576cd0954\") "
May 20 14:13:33.540578 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.540541    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fc80aa-7953-49f6-9d9d-3b3576cd0954-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "f9fc80aa-7953-49f6-9d9d-3b3576cd0954" (UID: "f9fc80aa-7953-49f6-9d9d-3b3576cd0954"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:13:33.542524 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.542501    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fc80aa-7953-49f6-9d9d-3b3576cd0954-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f9fc80aa-7953-49f6-9d9d-3b3576cd0954" (UID: "f9fc80aa-7953-49f6-9d9d-3b3576cd0954"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:13:33.641530 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.641496    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9fc80aa-7953-49f6-9d9d-3b3576cd0954-proxy-tls\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:13:33.641530 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.641530    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fc80aa-7953-49f6-9d9d-3b3576cd0954-openshift-service-ca-bundle\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:13:33.713918 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.713878    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm"]
May 20 14:13:33.714257 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.714243    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25bf4864-4618-4763-8476-04e940c2ed2c" containerName="kserve-container"
May 20 14:13:33.714301 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.714258    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="25bf4864-4618-4763-8476-04e940c2ed2c" containerName="kserve-container"
May 20 14:13:33.714301 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.714271    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fc80aa-7953-49f6-9d9d-3b3576cd0954" containerName="sequence-graph-cdd1d"
May 20 14:13:33.714301 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.714276    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fc80aa-7953-49f6-9d9d-3b3576cd0954" containerName="sequence-graph-cdd1d"
May 20 14:13:33.714301 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.714288    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25bf4864-4618-4763-8476-04e940c2ed2c" containerName="kube-rbac-proxy"
May 20 14:13:33.714301 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.714293    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="25bf4864-4618-4763-8476-04e940c2ed2c" containerName="kube-rbac-proxy"
May 20 14:13:33.714456 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.714367    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="25bf4864-4618-4763-8476-04e940c2ed2c" containerName="kube-rbac-proxy"
May 20 14:13:33.714456 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.714387    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="f9fc80aa-7953-49f6-9d9d-3b3576cd0954" containerName="sequence-graph-cdd1d"
May 20 14:13:33.714456 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.714396    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="25bf4864-4618-4763-8476-04e940c2ed2c" containerName="kserve-container"
May 20 14:13:33.717302 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.717276    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm"
May 20 14:13:33.719564 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.719545    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-15cdb-serving-cert\""
May 20 14:13:33.719564 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.719554    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-15cdb-kube-rbac-proxy-sar-config\""
May 20 14:13:33.729075 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.729046    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm"]
May 20 14:13:33.843413 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.843337    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6304b107-7353-4d51-a2c4-7880f1998511-openshift-service-ca-bundle\") pod \"ensemble-graph-15cdb-68844c9fc9-dsxnm\" (UID: \"6304b107-7353-4d51-a2c4-7880f1998511\") " pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm"
May 20 14:13:33.843413 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.843391    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6304b107-7353-4d51-a2c4-7880f1998511-proxy-tls\") pod \"ensemble-graph-15cdb-68844c9fc9-dsxnm\" (UID: \"6304b107-7353-4d51-a2c4-7880f1998511\") " pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm"
May 20 14:13:33.944892 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.944845    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6304b107-7353-4d51-a2c4-7880f1998511-openshift-service-ca-bundle\") pod \"ensemble-graph-15cdb-68844c9fc9-dsxnm\" (UID: \"6304b107-7353-4d51-a2c4-7880f1998511\") " pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm"
May 20 14:13:33.945048 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.944905    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6304b107-7353-4d51-a2c4-7880f1998511-proxy-tls\") pod \"ensemble-graph-15cdb-68844c9fc9-dsxnm\" (UID: \"6304b107-7353-4d51-a2c4-7880f1998511\") " pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm"
May 20 14:13:33.945048 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:13:33.945043    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-15cdb-serving-cert: secret "ensemble-graph-15cdb-serving-cert" not found
May 20 14:13:33.945140 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:13:33.945107    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6304b107-7353-4d51-a2c4-7880f1998511-proxy-tls podName:6304b107-7353-4d51-a2c4-7880f1998511 nodeName:}" failed. No retries permitted until 2026-05-20 14:13:34.445087552 +0000 UTC m=+1381.682704178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6304b107-7353-4d51-a2c4-7880f1998511-proxy-tls") pod "ensemble-graph-15cdb-68844c9fc9-dsxnm" (UID: "6304b107-7353-4d51-a2c4-7880f1998511") : secret "ensemble-graph-15cdb-serving-cert" not found
May 20 14:13:33.945494 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:33.945472    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6304b107-7353-4d51-a2c4-7880f1998511-openshift-service-ca-bundle\") pod \"ensemble-graph-15cdb-68844c9fc9-dsxnm\" (UID: \"6304b107-7353-4d51-a2c4-7880f1998511\") " pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm"
May 20 14:13:34.136357 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:34.136270    2570 generic.go:358] "Generic (PLEG): container finished" podID="f9fc80aa-7953-49f6-9d9d-3b3576cd0954" containerID="450e8815d4665fde7b8b44a1dbdb3208cbb8476dea99a12940681b945632a0ba" exitCode=0
May 20 14:13:34.136357 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:34.136347    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb"
May 20 14:13:34.136556 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:34.136360    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb" event={"ID":"f9fc80aa-7953-49f6-9d9d-3b3576cd0954","Type":"ContainerDied","Data":"450e8815d4665fde7b8b44a1dbdb3208cbb8476dea99a12940681b945632a0ba"}
May 20 14:13:34.136556 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:34.136408    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb" event={"ID":"f9fc80aa-7953-49f6-9d9d-3b3576cd0954","Type":"ContainerDied","Data":"c501efcdbc8bc35ffa5baf72115f681dad26574eff03d443590db866002d4c41"}
May 20 14:13:34.136556 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:34.136446    2570 scope.go:117] "RemoveContainer" containerID="450e8815d4665fde7b8b44a1dbdb3208cbb8476dea99a12940681b945632a0ba"
May 20 14:13:34.144844 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:34.144819    2570 scope.go:117] "RemoveContainer" containerID="450e8815d4665fde7b8b44a1dbdb3208cbb8476dea99a12940681b945632a0ba"
May 20 14:13:34.145118 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:13:34.145101    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"450e8815d4665fde7b8b44a1dbdb3208cbb8476dea99a12940681b945632a0ba\": container with ID starting with 450e8815d4665fde7b8b44a1dbdb3208cbb8476dea99a12940681b945632a0ba not found: ID does not exist" containerID="450e8815d4665fde7b8b44a1dbdb3208cbb8476dea99a12940681b945632a0ba"
May 20 14:13:34.145178 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:34.145126    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450e8815d4665fde7b8b44a1dbdb3208cbb8476dea99a12940681b945632a0ba"} err="failed to get container status \"450e8815d4665fde7b8b44a1dbdb3208cbb8476dea99a12940681b945632a0ba\": rpc error: code = NotFound desc = could not find container \"450e8815d4665fde7b8b44a1dbdb3208cbb8476dea99a12940681b945632a0ba\": container with ID starting with 450e8815d4665fde7b8b44a1dbdb3208cbb8476dea99a12940681b945632a0ba not found: ID does not exist"
May 20 14:13:34.162178 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:34.162143    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb"]
May 20 14:13:34.167571 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:34.167543    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-cdd1d-84988c9c55-pflbb"]
May 20 14:13:34.449194 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:34.449099    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6304b107-7353-4d51-a2c4-7880f1998511-proxy-tls\") pod \"ensemble-graph-15cdb-68844c9fc9-dsxnm\" (UID: \"6304b107-7353-4d51-a2c4-7880f1998511\") " pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm"
May 20 14:13:34.451806 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:34.451778    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6304b107-7353-4d51-a2c4-7880f1998511-proxy-tls\") pod \"ensemble-graph-15cdb-68844c9fc9-dsxnm\" (UID: \"6304b107-7353-4d51-a2c4-7880f1998511\") " pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm"
May 20 14:13:34.628549 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:34.628509    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm"
May 20 14:13:34.765280 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:34.765248    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm"]
May 20 14:13:34.768668 ip-10-0-136-75 kubenswrapper[2570]: W0520 14:13:34.768625    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6304b107_7353_4d51_a2c4_7880f1998511.slice/crio-24280dd7d2d450f034bb5922f82bd8946b5350e1bde79a280ab478687d6bd790 WatchSource:0}: Error finding container 24280dd7d2d450f034bb5922f82bd8946b5350e1bde79a280ab478687d6bd790: Status 404 returned error can't find the container with id 24280dd7d2d450f034bb5922f82bd8946b5350e1bde79a280ab478687d6bd790
May 20 14:13:35.141040 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:35.141004    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm" event={"ID":"6304b107-7353-4d51-a2c4-7880f1998511","Type":"ContainerStarted","Data":"1252c8062414ed37c9f420838d810aad9fdcb5e6900f7827dbc6336b6d297c6c"}
May 20 14:13:35.141040 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:35.141042    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm" event={"ID":"6304b107-7353-4d51-a2c4-7880f1998511","Type":"ContainerStarted","Data":"24280dd7d2d450f034bb5922f82bd8946b5350e1bde79a280ab478687d6bd790"}
May 20 14:13:35.141306 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:35.141138    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm"
May 20 14:13:35.158829 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:35.158784    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm" podStartSLOduration=2.158771391 podStartE2EDuration="2.158771391s" podCreationTimestamp="2026-05-20 14:13:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:13:35.157747185 +0000 UTC m=+1382.395363833" watchObservedRunningTime="2026-05-20 14:13:35.158771391 +0000 UTC m=+1382.396388067"
May 20 14:13:35.363730 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:35.363694    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9fc80aa-7953-49f6-9d9d-3b3576cd0954" path="/var/lib/kubelet/pods/f9fc80aa-7953-49f6-9d9d-3b3576cd0954/volumes"
May 20 14:13:41.151197 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:41.151170    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm"
May 20 14:13:42.044462 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:42.044423    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg" podUID="677266f2-8375-4dcf-a101-0160c6f8f5b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 14:13:43.781835 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:43.781800    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm"]
May 20 14:13:43.782275 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:43.782053    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm" podUID="6304b107-7353-4d51-a2c4-7880f1998511" containerName="ensemble-graph-15cdb" containerID="cri-o://1252c8062414ed37c9f420838d810aad9fdcb5e6900f7827dbc6336b6d297c6c" gracePeriod=30
May 20 14:13:44.004036 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:44.003996    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"]
May 20 14:13:44.004440 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:44.004388    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp" podUID="e93a15ed-cdfb-4bb2-b08f-935716638b5a" containerName="kserve-container" containerID="cri-o://0c1c5f2a8b64a287143afaf823441f95a446e1433df4946a176e7413ad775ceb" gracePeriod=30
May 20 14:13:44.004627 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:44.004577    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp" podUID="e93a15ed-cdfb-4bb2-b08f-935716638b5a" containerName="kube-rbac-proxy" containerID="cri-o://093bd16fc7a0ad6ef18e933600243923e6034bdd11490ae29906a421ffef9e2d" gracePeriod=30
May 20 14:13:44.074743 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:44.074665    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"]
May 20 14:13:44.077661 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:44.077641    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"
May 20 14:13:44.080735 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:44.080712    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ecafe-predictor-serving-cert\""
May 20 14:13:44.080917 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:44.080899    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ecafe-kube-rbac-proxy-sar-config\""
May 20 14:13:44.093022 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:44.092992    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"]
May 20 14:13:44.131815 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:44.131782    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xt84\" (UniqueName: \"kubernetes.io/projected/c82902d1-af79-484e-a67e-1f0ed4b08fc9-kube-api-access-4xt84\") pod \"error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4\" (UID: \"c82902d1-af79-484e-a67e-1f0ed4b08fc9\") " pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"
May 20 14:13:44.131969 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:44.131821    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-ecafe-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c82902d1-af79-484e-a67e-1f0ed4b08fc9-error-404-isvc-ecafe-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4\" (UID: \"c82902d1-af79-484e-a67e-1f0ed4b08fc9\") " pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"
May 20 14:13:44.131969 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:44.131939    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c82902d1-af79-484e-a67e-1f0ed4b08fc9-proxy-tls\") pod \"error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4\" (UID: \"c82902d1-af79-484e-a67e-1f0ed4b08fc9\") " pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"
May 20 14:13:44.175035 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:44.174993    2570 generic.go:358] "Generic (PLEG): container finished" podID="e93a15ed-cdfb-4bb2-b08f-935716638b5a" containerID="093bd16fc7a0ad6ef18e933600243923e6034bdd11490ae29906a421ffef9e2d" exitCode=2
May 20 14:13:44.175221 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:44.175080    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp" event={"ID":"e93a15ed-cdfb-4bb2-b08f-935716638b5a","Type":"ContainerDied","Data":"093bd16fc7a0ad6ef18e933600243923e6034bdd11490ae29906a421ffef9e2d"}
May 20 14:13:44.232808 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:44.232771    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c82902d1-af79-484e-a67e-1f0ed4b08fc9-proxy-tls\") pod \"error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4\" (UID: \"c82902d1-af79-484e-a67e-1f0ed4b08fc9\") " pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"
May 20 14:13:44.232981 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:44.232864    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4xt84\" (UniqueName: \"kubernetes.io/projected/c82902d1-af79-484e-a67e-1f0ed4b08fc9-kube-api-access-4xt84\") pod \"error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4\" (UID: \"c82902d1-af79-484e-a67e-1f0ed4b08fc9\") " pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"
May 20 14:13:44.232981 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:44.232892    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-ecafe-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c82902d1-af79-484e-a67e-1f0ed4b08fc9-error-404-isvc-ecafe-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4\" (UID: \"c82902d1-af79-484e-a67e-1f0ed4b08fc9\") " pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"
May 20 14:13:44.233560 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:44.233541    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-ecafe-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c82902d1-af79-484e-a67e-1f0ed4b08fc9-error-404-isvc-ecafe-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4\" (UID: \"c82902d1-af79-484e-a67e-1f0ed4b08fc9\") " pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"
May 20 14:13:44.235365 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:44.235346    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c82902d1-af79-484e-a67e-1f0ed4b08fc9-proxy-tls\") pod \"error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4\" (UID: \"c82902d1-af79-484e-a67e-1f0ed4b08fc9\") " pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"
May 20 14:13:44.243923 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:44.243896    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xt84\" (UniqueName: \"kubernetes.io/projected/c82902d1-af79-484e-a67e-1f0ed4b08fc9-kube-api-access-4xt84\") pod \"error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4\" (UID: \"c82902d1-af79-484e-a67e-1f0ed4b08fc9\") " pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"
May 20 14:13:44.387970 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:44.387891    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"
May 20 14:13:44.528360 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:44.528335    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"]
May 20 14:13:44.530806 ip-10-0-136-75 kubenswrapper[2570]: W0520 14:13:44.530780    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82902d1_af79_484e_a67e_1f0ed4b08fc9.slice/crio-a5a9b4512bd96503e8fd4dd4acdc6ff4cf3d1e977441b46ed2c06953b207889e WatchSource:0}: Error finding container a5a9b4512bd96503e8fd4dd4acdc6ff4cf3d1e977441b46ed2c06953b207889e: Status 404 returned error can't find the container with id a5a9b4512bd96503e8fd4dd4acdc6ff4cf3d1e977441b46ed2c06953b207889e
May 20 14:13:45.181034 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:45.180994    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4" event={"ID":"c82902d1-af79-484e-a67e-1f0ed4b08fc9","Type":"ContainerStarted","Data":"262fc6817d94dda942fd48375e3c90fe6c28f0127a7b112216983ea622a2484a"}
May 20 14:13:45.181034 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:45.181039    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4" event={"ID":"c82902d1-af79-484e-a67e-1f0ed4b08fc9","Type":"ContainerStarted","Data":"a0b7ea4d2c83a0f41e65d7a252717350071d2c8f227a55ba36c56e54ce01179a"}
May 20 14:13:45.181478 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:45.181054    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4" event={"ID":"c82902d1-af79-484e-a67e-1f0ed4b08fc9","Type":"ContainerStarted","Data":"a5a9b4512bd96503e8fd4dd4acdc6ff4cf3d1e977441b46ed2c06953b207889e"}
May 20 14:13:45.181478 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:45.181140    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"
May 20 14:13:45.204059 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:45.203990    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4" podStartSLOduration=1.203974668 podStartE2EDuration="1.203974668s" podCreationTimestamp="2026-05-20 14:13:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:13:45.203697929 +0000 UTC m=+1392.441314578" watchObservedRunningTime="2026-05-20 14:13:45.203974668 +0000 UTC m=+1392.441591372"
May 20 14:13:45.921046 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:45.921000    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp" podUID="e93a15ed-cdfb-4bb2-b08f-935716638b5a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
May 20 14:13:46.149116 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:46.149076    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm" podUID="6304b107-7353-4d51-a2c4-7880f1998511" containerName="ensemble-graph-15cdb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:13:46.184988 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:46.184916    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"
May 20 14:13:46.186221 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:46.186191    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4" podUID="c82902d1-af79-484e-a67e-1f0ed4b08fc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 20 14:13:47.188547 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:47.188506    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4" podUID="c82902d1-af79-484e-a67e-1f0ed4b08fc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 20 14:13:47.558939 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:47.558912    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"
May 20 14:13:47.664592 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:47.664553    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxtbt\" (UniqueName: \"kubernetes.io/projected/e93a15ed-cdfb-4bb2-b08f-935716638b5a-kube-api-access-mxtbt\") pod \"e93a15ed-cdfb-4bb2-b08f-935716638b5a\" (UID: \"e93a15ed-cdfb-4bb2-b08f-935716638b5a\") "
May 20 14:13:47.664819 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:47.664638    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e93a15ed-cdfb-4bb2-b08f-935716638b5a-proxy-tls\") pod \"e93a15ed-cdfb-4bb2-b08f-935716638b5a\" (UID: \"e93a15ed-cdfb-4bb2-b08f-935716638b5a\") "
May 20 14:13:47.664819 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:47.664713    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-15cdb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e93a15ed-cdfb-4bb2-b08f-935716638b5a-error-404-isvc-15cdb-kube-rbac-proxy-sar-config\") pod \"e93a15ed-cdfb-4bb2-b08f-935716638b5a\" (UID: \"e93a15ed-cdfb-4bb2-b08f-935716638b5a\") "
May 20 14:13:47.665055 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:47.665028    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e93a15ed-cdfb-4bb2-b08f-935716638b5a-error-404-isvc-15cdb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-15cdb-kube-rbac-proxy-sar-config") pod "e93a15ed-cdfb-4bb2-b08f-935716638b5a" (UID: "e93a15ed-cdfb-4bb2-b08f-935716638b5a"). InnerVolumeSpecName "error-404-isvc-15cdb-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:13:47.667018 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:47.666994    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e93a15ed-cdfb-4bb2-b08f-935716638b5a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e93a15ed-cdfb-4bb2-b08f-935716638b5a" (UID: "e93a15ed-cdfb-4bb2-b08f-935716638b5a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:13:47.667115 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:47.667020    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e93a15ed-cdfb-4bb2-b08f-935716638b5a-kube-api-access-mxtbt" (OuterVolumeSpecName: "kube-api-access-mxtbt") pod "e93a15ed-cdfb-4bb2-b08f-935716638b5a" (UID: "e93a15ed-cdfb-4bb2-b08f-935716638b5a"). InnerVolumeSpecName "kube-api-access-mxtbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:13:47.765635 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:47.765518    2570 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-15cdb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e93a15ed-cdfb-4bb2-b08f-935716638b5a-error-404-isvc-15cdb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:13:47.765635 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:47.765555    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxtbt\" (UniqueName: \"kubernetes.io/projected/e93a15ed-cdfb-4bb2-b08f-935716638b5a-kube-api-access-mxtbt\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:13:47.765635 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:47.765565    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e93a15ed-cdfb-4bb2-b08f-935716638b5a-proxy-tls\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:13:48.193149 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:48.193119    2570 generic.go:358] "Generic (PLEG): container finished" podID="e93a15ed-cdfb-4bb2-b08f-935716638b5a" containerID="0c1c5f2a8b64a287143afaf823441f95a446e1433df4946a176e7413ad775ceb" exitCode=0
May 20 14:13:48.193563 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:48.193192    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"
May 20 14:13:48.193563 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:48.193202    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp" event={"ID":"e93a15ed-cdfb-4bb2-b08f-935716638b5a","Type":"ContainerDied","Data":"0c1c5f2a8b64a287143afaf823441f95a446e1433df4946a176e7413ad775ceb"}
May 20 14:13:48.193563 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:48.193243    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp" event={"ID":"e93a15ed-cdfb-4bb2-b08f-935716638b5a","Type":"ContainerDied","Data":"93bcd0cce570e8d02cda7070eeef52e948f0699336a8864e4897bea1e1440273"}
May 20 14:13:48.193563 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:48.193259    2570 scope.go:117] "RemoveContainer" containerID="093bd16fc7a0ad6ef18e933600243923e6034bdd11490ae29906a421ffef9e2d"
May 20 14:13:48.201754 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:48.201735    2570 scope.go:117] "RemoveContainer" containerID="0c1c5f2a8b64a287143afaf823441f95a446e1433df4946a176e7413ad775ceb"
May 20 14:13:48.209252 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:48.209234    2570 scope.go:117] "RemoveContainer" containerID="093bd16fc7a0ad6ef18e933600243923e6034bdd11490ae29906a421ffef9e2d"
May 20 14:13:48.209537 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:13:48.209518    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"093bd16fc7a0ad6ef18e933600243923e6034bdd11490ae29906a421ffef9e2d\": container with ID starting with 093bd16fc7a0ad6ef18e933600243923e6034bdd11490ae29906a421ffef9e2d not found: ID does not exist" containerID="093bd16fc7a0ad6ef18e933600243923e6034bdd11490ae29906a421ffef9e2d"
May 20 14:13:48.209603 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:48.209545    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"093bd16fc7a0ad6ef18e933600243923e6034bdd11490ae29906a421ffef9e2d"} err="failed to get container status \"093bd16fc7a0ad6ef18e933600243923e6034bdd11490ae29906a421ffef9e2d\": rpc error: code = NotFound desc = could not find container \"093bd16fc7a0ad6ef18e933600243923e6034bdd11490ae29906a421ffef9e2d\": container with ID starting with 093bd16fc7a0ad6ef18e933600243923e6034bdd11490ae29906a421ffef9e2d not found: ID does not exist"
May 20 14:13:48.209603 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:48.209563    2570 scope.go:117] "RemoveContainer" containerID="0c1c5f2a8b64a287143afaf823441f95a446e1433df4946a176e7413ad775ceb"
May 20 14:13:48.209817 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:13:48.209800    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c1c5f2a8b64a287143afaf823441f95a446e1433df4946a176e7413ad775ceb\": container with ID starting with 0c1c5f2a8b64a287143afaf823441f95a446e1433df4946a176e7413ad775ceb not found: ID does not exist" containerID="0c1c5f2a8b64a287143afaf823441f95a446e1433df4946a176e7413ad775ceb"
May 20 14:13:48.209858 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:48.209824    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c1c5f2a8b64a287143afaf823441f95a446e1433df4946a176e7413ad775ceb"} err="failed to get container status \"0c1c5f2a8b64a287143afaf823441f95a446e1433df4946a176e7413ad775ceb\": rpc error: code = NotFound desc = could not find container \"0c1c5f2a8b64a287143afaf823441f95a446e1433df4946a176e7413ad775ceb\": container with ID starting with 0c1c5f2a8b64a287143afaf823441f95a446e1433df4946a176e7413ad775ceb not found: ID does not exist"
May 20 14:13:48.216337 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:48.216298    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"]
May 20 14:13:48.220792 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:48.220767    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-15cdb-predictor-7c646df98f-sjwvp"]
May 20 14:13:49.364230 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:49.364199    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e93a15ed-cdfb-4bb2-b08f-935716638b5a" path="/var/lib/kubelet/pods/e93a15ed-cdfb-4bb2-b08f-935716638b5a/volumes"
May 20 14:13:51.149853 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:51.149814    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm" podUID="6304b107-7353-4d51-a2c4-7880f1998511" containerName="ensemble-graph-15cdb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:13:52.045466 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:52.045435    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"
May 20 14:13:52.192804 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:52.192778    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"
May 20 14:13:52.193449 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:52.193408    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4" podUID="c82902d1-af79-484e-a67e-1f0ed4b08fc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 20 14:13:56.149784 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:56.149741    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm" podUID="6304b107-7353-4d51-a2c4-7880f1998511" containerName="ensemble-graph-15cdb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:13:56.150232 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:13:56.149850    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm"
May 20 14:14:01.148844 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:01.148802    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm" podUID="6304b107-7353-4d51-a2c4-7880f1998511" containerName="ensemble-graph-15cdb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:14:02.194308 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:02.194263    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4" podUID="c82902d1-af79-484e-a67e-1f0ed4b08fc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 20 14:14:03.432224 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:03.432190    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf"]
May 20 14:14:03.432668 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:03.432555    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e93a15ed-cdfb-4bb2-b08f-935716638b5a" containerName="kserve-container"
May 20 14:14:03.432668 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:03.432566    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e93a15ed-cdfb-4bb2-b08f-935716638b5a" containerName="kserve-container"
May 20 14:14:03.432668 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:03.432577    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e93a15ed-cdfb-4bb2-b08f-935716638b5a" containerName="kube-rbac-proxy"
May 20 14:14:03.432668 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:03.432582    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e93a15ed-cdfb-4bb2-b08f-935716638b5a" containerName="kube-rbac-proxy"
May 20 14:14:03.432823 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:03.432681    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="e93a15ed-cdfb-4bb2-b08f-935716638b5a" containerName="kserve-container"
May 20 14:14:03.432823 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:03.432691    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="e93a15ed-cdfb-4bb2-b08f-935716638b5a" containerName="kube-rbac-proxy"
May 20 14:14:03.435682 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:03.435661    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf"
May 20 14:14:03.438143 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:03.438118    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-14a90-serving-cert\""
May 20 14:14:03.438222 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:03.438131    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-14a90-kube-rbac-proxy-sar-config\""
May 20 14:14:03.445716 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:03.445691    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf"]
May 20 14:14:03.507045 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:03.507006    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/faae5ee3-1a59-4460-86dd-a79d893701c1-proxy-tls\") pod \"sequence-graph-14a90-6db7cfb9f9-9d8xf\" (UID: \"faae5ee3-1a59-4460-86dd-a79d893701c1\") " pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf"
May 20 14:14:03.507230 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:03.507094    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/faae5ee3-1a59-4460-86dd-a79d893701c1-openshift-service-ca-bundle\") pod \"sequence-graph-14a90-6db7cfb9f9-9d8xf\" (UID: \"faae5ee3-1a59-4460-86dd-a79d893701c1\") " pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf"
May 20 14:14:03.608802 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:03.608762    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/faae5ee3-1a59-4460-86dd-a79d893701c1-proxy-tls\") pod \"sequence-graph-14a90-6db7cfb9f9-9d8xf\" (UID: \"faae5ee3-1a59-4460-86dd-a79d893701c1\") " pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf"
May 20 14:14:03.608983 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:03.608814    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/faae5ee3-1a59-4460-86dd-a79d893701c1-openshift-service-ca-bundle\") pod \"sequence-graph-14a90-6db7cfb9f9-9d8xf\" (UID: \"faae5ee3-1a59-4460-86dd-a79d893701c1\") " pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf"
May 20 14:14:03.608983 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:14:03.608924    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-14a90-serving-cert: secret "sequence-graph-14a90-serving-cert" not found
May 20 14:14:03.609074 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:14:03.609009    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faae5ee3-1a59-4460-86dd-a79d893701c1-proxy-tls podName:faae5ee3-1a59-4460-86dd-a79d893701c1 nodeName:}" failed. No retries permitted until 2026-05-20 14:14:04.108990296 +0000 UTC m=+1411.346606921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/faae5ee3-1a59-4460-86dd-a79d893701c1-proxy-tls") pod "sequence-graph-14a90-6db7cfb9f9-9d8xf" (UID: "faae5ee3-1a59-4460-86dd-a79d893701c1") : secret "sequence-graph-14a90-serving-cert" not found
May 20 14:14:03.609417 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:03.609398    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/faae5ee3-1a59-4460-86dd-a79d893701c1-openshift-service-ca-bundle\") pod \"sequence-graph-14a90-6db7cfb9f9-9d8xf\" (UID: \"faae5ee3-1a59-4460-86dd-a79d893701c1\") " pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf"
May 20 14:14:04.113223 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:04.113182    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/faae5ee3-1a59-4460-86dd-a79d893701c1-proxy-tls\") pod \"sequence-graph-14a90-6db7cfb9f9-9d8xf\" (UID: \"faae5ee3-1a59-4460-86dd-a79d893701c1\") " pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf"
May 20 14:14:04.115847 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:04.115824    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/faae5ee3-1a59-4460-86dd-a79d893701c1-proxy-tls\") pod \"sequence-graph-14a90-6db7cfb9f9-9d8xf\" (UID: \"faae5ee3-1a59-4460-86dd-a79d893701c1\") " pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf"
May 20 14:14:04.346968 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:04.346908    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf"
May 20 14:14:04.481849 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:04.481821    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf"]
May 20 14:14:04.484118 ip-10-0-136-75 kubenswrapper[2570]: W0520 14:14:04.484091    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaae5ee3_1a59_4460_86dd_a79d893701c1.slice/crio-aaf9b56d41c50d3afe4637d333a9b7247f13d7ab22b830815dcf82312c82c344 WatchSource:0}: Error finding container aaf9b56d41c50d3afe4637d333a9b7247f13d7ab22b830815dcf82312c82c344: Status 404 returned error can't find the container with id aaf9b56d41c50d3afe4637d333a9b7247f13d7ab22b830815dcf82312c82c344
May 20 14:14:05.253325 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:05.253292    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf" event={"ID":"faae5ee3-1a59-4460-86dd-a79d893701c1","Type":"ContainerStarted","Data":"d86c55d24646324575cae8909d828eab462a21e647e2bbb24443e68a18322b24"}
May 20 14:14:05.253325 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:05.253330    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf" event={"ID":"faae5ee3-1a59-4460-86dd-a79d893701c1","Type":"ContainerStarted","Data":"aaf9b56d41c50d3afe4637d333a9b7247f13d7ab22b830815dcf82312c82c344"}
May 20 14:14:05.253544 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:05.253403    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf"
May 20 14:14:05.271450 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:05.271396    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf" podStartSLOduration=2.271382484 podStartE2EDuration="2.271382484s" podCreationTimestamp="2026-05-20 14:14:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:14:05.270380838 +0000 UTC m=+1412.507997489" watchObservedRunningTime="2026-05-20 14:14:05.271382484 +0000 UTC m=+1412.508999155"
May 20 14:14:06.149687 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:06.149645    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm" podUID="6304b107-7353-4d51-a2c4-7880f1998511" containerName="ensemble-graph-15cdb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:14:11.149153 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:11.149102    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm" podUID="6304b107-7353-4d51-a2c4-7880f1998511" containerName="ensemble-graph-15cdb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:14:11.262876 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:11.262848    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf"
May 20 14:14:12.193353 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:12.193315    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4" podUID="c82902d1-af79-484e-a67e-1f0ed4b08fc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 20 14:14:13.536643 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:13.536580    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf"]
May 20 14:14:13.537116 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:13.536872    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf" podUID="faae5ee3-1a59-4460-86dd-a79d893701c1" containerName="sequence-graph-14a90" containerID="cri-o://d86c55d24646324575cae8909d828eab462a21e647e2bbb24443e68a18322b24" gracePeriod=30
May 20 14:14:13.633526 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:13.633488    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"]
May 20 14:14:13.633926 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:13.633879    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg" podUID="677266f2-8375-4dcf-a101-0160c6f8f5b4" containerName="kserve-container" containerID="cri-o://4605bb3d807b86027680bc49e7fc40743c28cd3fdd03302a2e80ab2101794de4" gracePeriod=30
May 20 14:14:13.634027 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:13.633967    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg" podUID="677266f2-8375-4dcf-a101-0160c6f8f5b4" containerName="kube-rbac-proxy" containerID="cri-o://e1e398541b2fa91372dc5a3236d371cc258ac181839d44162982022181010609" gracePeriod=30
May 20 14:14:13.811418 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:13.811384    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"]
May 20 14:14:13.814897 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:13.814879    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"
May 20 14:14:13.817220 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:13.817189    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-734fc-predictor-serving-cert\""
May 20 14:14:13.817335 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:13.817194    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-734fc-kube-rbac-proxy-sar-config\""
May 20 14:14:13.841123 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:13.841090    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"]
May 20 14:14:13.898272 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:13.898233    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-734fc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bb62959-33c7-4cf3-824a-616ee8f239e2-error-404-isvc-734fc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-734fc-predictor-5995d7675c-86m5w\" (UID: \"7bb62959-33c7-4cf3-824a-616ee8f239e2\") " pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"
May 20 14:14:13.898415 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:13.898294    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mlp5\" (UniqueName: \"kubernetes.io/projected/7bb62959-33c7-4cf3-824a-616ee8f239e2-kube-api-access-2mlp5\") pod \"error-404-isvc-734fc-predictor-5995d7675c-86m5w\" (UID: \"7bb62959-33c7-4cf3-824a-616ee8f239e2\") " pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"
May 20 14:14:13.898415 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:13.898406    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bb62959-33c7-4cf3-824a-616ee8f239e2-proxy-tls\") pod \"error-404-isvc-734fc-predictor-5995d7675c-86m5w\" (UID: \"7bb62959-33c7-4cf3-824a-616ee8f239e2\") " pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"
May 20 14:14:13.972386 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:13.972359    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm"
May 20 14:14:13.999675 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:13.999640    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bb62959-33c7-4cf3-824a-616ee8f239e2-proxy-tls\") pod \"error-404-isvc-734fc-predictor-5995d7675c-86m5w\" (UID: \"7bb62959-33c7-4cf3-824a-616ee8f239e2\") " pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"
May 20 14:14:13.999852 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:13.999738    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-734fc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bb62959-33c7-4cf3-824a-616ee8f239e2-error-404-isvc-734fc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-734fc-predictor-5995d7675c-86m5w\" (UID: \"7bb62959-33c7-4cf3-824a-616ee8f239e2\") " pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"
May 20 14:14:13.999852 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:13.999767    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2mlp5\" (UniqueName: \"kubernetes.io/projected/7bb62959-33c7-4cf3-824a-616ee8f239e2-kube-api-access-2mlp5\") pod \"error-404-isvc-734fc-predictor-5995d7675c-86m5w\" (UID: \"7bb62959-33c7-4cf3-824a-616ee8f239e2\") " pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"
May 20 14:14:13.999852 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:14:13.999774    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-734fc-predictor-serving-cert: secret "error-404-isvc-734fc-predictor-serving-cert" not found
May 20 14:14:13.999852 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:14:13.999837    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb62959-33c7-4cf3-824a-616ee8f239e2-proxy-tls podName:7bb62959-33c7-4cf3-824a-616ee8f239e2 nodeName:}" failed. No retries permitted until 2026-05-20 14:14:14.499815931 +0000 UTC m=+1421.737432557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7bb62959-33c7-4cf3-824a-616ee8f239e2-proxy-tls") pod "error-404-isvc-734fc-predictor-5995d7675c-86m5w" (UID: "7bb62959-33c7-4cf3-824a-616ee8f239e2") : secret "error-404-isvc-734fc-predictor-serving-cert" not found
May 20 14:14:14.000400 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.000380    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-734fc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bb62959-33c7-4cf3-824a-616ee8f239e2-error-404-isvc-734fc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-734fc-predictor-5995d7675c-86m5w\" (UID: \"7bb62959-33c7-4cf3-824a-616ee8f239e2\") " pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"
May 20 14:14:14.022793 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.022758    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mlp5\" (UniqueName: \"kubernetes.io/projected/7bb62959-33c7-4cf3-824a-616ee8f239e2-kube-api-access-2mlp5\") pod \"error-404-isvc-734fc-predictor-5995d7675c-86m5w\" (UID: \"7bb62959-33c7-4cf3-824a-616ee8f239e2\") " pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"
May 20 14:14:14.100365 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.100279    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6304b107-7353-4d51-a2c4-7880f1998511-openshift-service-ca-bundle\") pod \"6304b107-7353-4d51-a2c4-7880f1998511\" (UID: \"6304b107-7353-4d51-a2c4-7880f1998511\") "
May 20 14:14:14.100535 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.100390    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6304b107-7353-4d51-a2c4-7880f1998511-proxy-tls\") pod \"6304b107-7353-4d51-a2c4-7880f1998511\" (UID: \"6304b107-7353-4d51-a2c4-7880f1998511\") "
May 20 14:14:14.100718 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.100688    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6304b107-7353-4d51-a2c4-7880f1998511-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6304b107-7353-4d51-a2c4-7880f1998511" (UID: "6304b107-7353-4d51-a2c4-7880f1998511"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:14:14.102793 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.102769    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6304b107-7353-4d51-a2c4-7880f1998511-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6304b107-7353-4d51-a2c4-7880f1998511" (UID: "6304b107-7353-4d51-a2c4-7880f1998511"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:14:14.202019 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.201983    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6304b107-7353-4d51-a2c4-7880f1998511-proxy-tls\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:14:14.202019 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.202010    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6304b107-7353-4d51-a2c4-7880f1998511-openshift-service-ca-bundle\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:14:14.290450 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.290413    2570 generic.go:358] "Generic (PLEG): container finished" podID="6304b107-7353-4d51-a2c4-7880f1998511" containerID="1252c8062414ed37c9f420838d810aad9fdcb5e6900f7827dbc6336b6d297c6c" exitCode=0
May 20 14:14:14.290735 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.290474    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm"
May 20 14:14:14.290735 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.290489    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm" event={"ID":"6304b107-7353-4d51-a2c4-7880f1998511","Type":"ContainerDied","Data":"1252c8062414ed37c9f420838d810aad9fdcb5e6900f7827dbc6336b6d297c6c"}
May 20 14:14:14.290735 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.290521    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm" event={"ID":"6304b107-7353-4d51-a2c4-7880f1998511","Type":"ContainerDied","Data":"24280dd7d2d450f034bb5922f82bd8946b5350e1bde79a280ab478687d6bd790"}
May 20 14:14:14.290735 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.290536    2570 scope.go:117] "RemoveContainer" containerID="1252c8062414ed37c9f420838d810aad9fdcb5e6900f7827dbc6336b6d297c6c"
May 20 14:14:14.292235 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.292208    2570 generic.go:358] "Generic (PLEG): container finished" podID="677266f2-8375-4dcf-a101-0160c6f8f5b4" containerID="e1e398541b2fa91372dc5a3236d371cc258ac181839d44162982022181010609" exitCode=2
May 20 14:14:14.292358 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.292273    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg" event={"ID":"677266f2-8375-4dcf-a101-0160c6f8f5b4","Type":"ContainerDied","Data":"e1e398541b2fa91372dc5a3236d371cc258ac181839d44162982022181010609"}
May 20 14:14:14.299552 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.299529    2570 scope.go:117] "RemoveContainer" containerID="1252c8062414ed37c9f420838d810aad9fdcb5e6900f7827dbc6336b6d297c6c"
May 20 14:14:14.299859 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:14:14.299839    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1252c8062414ed37c9f420838d810aad9fdcb5e6900f7827dbc6336b6d297c6c\": container with ID starting with 1252c8062414ed37c9f420838d810aad9fdcb5e6900f7827dbc6336b6d297c6c not found: ID does not exist" containerID="1252c8062414ed37c9f420838d810aad9fdcb5e6900f7827dbc6336b6d297c6c"
May 20 14:14:14.299913 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.299867    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1252c8062414ed37c9f420838d810aad9fdcb5e6900f7827dbc6336b6d297c6c"} err="failed to get container status \"1252c8062414ed37c9f420838d810aad9fdcb5e6900f7827dbc6336b6d297c6c\": rpc error: code = NotFound desc = could not find container \"1252c8062414ed37c9f420838d810aad9fdcb5e6900f7827dbc6336b6d297c6c\": container with ID starting with 1252c8062414ed37c9f420838d810aad9fdcb5e6900f7827dbc6336b6d297c6c not found: ID does not exist"
May 20 14:14:14.313853 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.313821    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm"]
May 20 14:14:14.319440 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.319416    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-15cdb-68844c9fc9-dsxnm"]
May 20 14:14:14.505321 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.505235    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bb62959-33c7-4cf3-824a-616ee8f239e2-proxy-tls\") pod \"error-404-isvc-734fc-predictor-5995d7675c-86m5w\" (UID: \"7bb62959-33c7-4cf3-824a-616ee8f239e2\") " pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"
May 20 14:14:14.507803 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.507786    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bb62959-33c7-4cf3-824a-616ee8f239e2-proxy-tls\") pod \"error-404-isvc-734fc-predictor-5995d7675c-86m5w\" (UID: \"7bb62959-33c7-4cf3-824a-616ee8f239e2\") " pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"
May 20 14:14:14.752118 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.752085    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"
May 20 14:14:14.887148 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:14.887075    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"]
May 20 14:14:14.889494 ip-10-0-136-75 kubenswrapper[2570]: W0520 14:14:14.889460    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb62959_33c7_4cf3_824a_616ee8f239e2.slice/crio-d5ff8aa537da36dc27cccd9a75291d97fe53fba0b2bb87bec6ace99b33f20caf WatchSource:0}: Error finding container d5ff8aa537da36dc27cccd9a75291d97fe53fba0b2bb87bec6ace99b33f20caf: Status 404 returned error can't find the container with id d5ff8aa537da36dc27cccd9a75291d97fe53fba0b2bb87bec6ace99b33f20caf
May 20 14:14:15.299376 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:15.299291    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w" event={"ID":"7bb62959-33c7-4cf3-824a-616ee8f239e2","Type":"ContainerStarted","Data":"1d4eb9d5e21f33468d91cbedb36c06b2fe1afd09615cbe34382c4790a83e9b34"}
May 20 14:14:15.299376 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:15.299334    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w" event={"ID":"7bb62959-33c7-4cf3-824a-616ee8f239e2","Type":"ContainerStarted","Data":"3b077e31529d7078dcc0f4d7557b123050863156e7d01674a9eab49c28a7ff1e"}
May 20 14:14:15.299376 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:15.299348    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w" event={"ID":"7bb62959-33c7-4cf3-824a-616ee8f239e2","Type":"ContainerStarted","Data":"d5ff8aa537da36dc27cccd9a75291d97fe53fba0b2bb87bec6ace99b33f20caf"}
May 20 14:14:15.299586 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:15.299450    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"
May 20 14:14:15.321372 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:15.321323    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w" podStartSLOduration=2.321308413 podStartE2EDuration="2.321308413s" podCreationTimestamp="2026-05-20 14:14:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:14:15.320221417 +0000 UTC m=+1422.557838081" watchObservedRunningTime="2026-05-20 14:14:15.321308413 +0000 UTC m=+1422.558925111"
May 20 14:14:15.363896 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:15.363861    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6304b107-7353-4d51-a2c4-7880f1998511" path="/var/lib/kubelet/pods/6304b107-7353-4d51-a2c4-7880f1998511/volumes"
May 20 14:14:16.260583 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:16.260542    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf" podUID="faae5ee3-1a59-4460-86dd-a79d893701c1" containerName="sequence-graph-14a90" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:14:16.303027 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:16.302993    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"
May 20 14:14:16.304297 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:16.304270    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w" podUID="7bb62959-33c7-4cf3-824a-616ee8f239e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
May 20 14:14:17.084379 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.084355    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"
May 20 14:14:17.228722 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.228636    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-14a90-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/677266f2-8375-4dcf-a101-0160c6f8f5b4-error-404-isvc-14a90-kube-rbac-proxy-sar-config\") pod \"677266f2-8375-4dcf-a101-0160c6f8f5b4\" (UID: \"677266f2-8375-4dcf-a101-0160c6f8f5b4\") "
May 20 14:14:17.228722 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.228717    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqwp7\" (UniqueName: \"kubernetes.io/projected/677266f2-8375-4dcf-a101-0160c6f8f5b4-kube-api-access-lqwp7\") pod \"677266f2-8375-4dcf-a101-0160c6f8f5b4\" (UID: \"677266f2-8375-4dcf-a101-0160c6f8f5b4\") "
May 20 14:14:17.228941 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.228744    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/677266f2-8375-4dcf-a101-0160c6f8f5b4-proxy-tls\") pod \"677266f2-8375-4dcf-a101-0160c6f8f5b4\" (UID: \"677266f2-8375-4dcf-a101-0160c6f8f5b4\") "
May 20 14:14:17.229002 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.228970    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/677266f2-8375-4dcf-a101-0160c6f8f5b4-error-404-isvc-14a90-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-14a90-kube-rbac-proxy-sar-config") pod "677266f2-8375-4dcf-a101-0160c6f8f5b4" (UID: "677266f2-8375-4dcf-a101-0160c6f8f5b4"). InnerVolumeSpecName "error-404-isvc-14a90-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:14:17.230940 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.230915    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677266f2-8375-4dcf-a101-0160c6f8f5b4-kube-api-access-lqwp7" (OuterVolumeSpecName: "kube-api-access-lqwp7") pod "677266f2-8375-4dcf-a101-0160c6f8f5b4" (UID: "677266f2-8375-4dcf-a101-0160c6f8f5b4"). InnerVolumeSpecName "kube-api-access-lqwp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:14:17.230940 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.230928    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/677266f2-8375-4dcf-a101-0160c6f8f5b4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "677266f2-8375-4dcf-a101-0160c6f8f5b4" (UID: "677266f2-8375-4dcf-a101-0160c6f8f5b4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:14:17.308256 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.308220    2570 generic.go:358] "Generic (PLEG): container finished" podID="677266f2-8375-4dcf-a101-0160c6f8f5b4" containerID="4605bb3d807b86027680bc49e7fc40743c28cd3fdd03302a2e80ab2101794de4" exitCode=0
May 20 14:14:17.308725 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.308303    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"
May 20 14:14:17.308725 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.308308    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg" event={"ID":"677266f2-8375-4dcf-a101-0160c6f8f5b4","Type":"ContainerDied","Data":"4605bb3d807b86027680bc49e7fc40743c28cd3fdd03302a2e80ab2101794de4"}
May 20 14:14:17.308725 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.308351    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg" event={"ID":"677266f2-8375-4dcf-a101-0160c6f8f5b4","Type":"ContainerDied","Data":"f8f87f65dab0a769fb04a5744ca4ee5f51db0300baa97e78227bc6db13dc084b"}
May 20 14:14:17.308725 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.308369    2570 scope.go:117] "RemoveContainer" containerID="e1e398541b2fa91372dc5a3236d371cc258ac181839d44162982022181010609"
May 20 14:14:17.309016 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.308987    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w" podUID="7bb62959-33c7-4cf3-824a-616ee8f239e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
May 20 14:14:17.317500 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.317484    2570 scope.go:117] "RemoveContainer" containerID="4605bb3d807b86027680bc49e7fc40743c28cd3fdd03302a2e80ab2101794de4"
May 20 14:14:17.325372 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.325354    2570 scope.go:117] "RemoveContainer" containerID="e1e398541b2fa91372dc5a3236d371cc258ac181839d44162982022181010609"
May 20 14:14:17.325663 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:14:17.325644    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1e398541b2fa91372dc5a3236d371cc258ac181839d44162982022181010609\": container with ID starting with e1e398541b2fa91372dc5a3236d371cc258ac181839d44162982022181010609 not found: ID does not exist" containerID="e1e398541b2fa91372dc5a3236d371cc258ac181839d44162982022181010609"
May 20 14:14:17.325708 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.325673    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1e398541b2fa91372dc5a3236d371cc258ac181839d44162982022181010609"} err="failed to get container status \"e1e398541b2fa91372dc5a3236d371cc258ac181839d44162982022181010609\": rpc error: code = NotFound desc = could not find container \"e1e398541b2fa91372dc5a3236d371cc258ac181839d44162982022181010609\": container with ID starting with e1e398541b2fa91372dc5a3236d371cc258ac181839d44162982022181010609 not found: ID does not exist"
May 20 14:14:17.325708 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.325689    2570 scope.go:117] "RemoveContainer" containerID="4605bb3d807b86027680bc49e7fc40743c28cd3fdd03302a2e80ab2101794de4"
May 20 14:14:17.325928 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:14:17.325912    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4605bb3d807b86027680bc49e7fc40743c28cd3fdd03302a2e80ab2101794de4\": container with ID starting with 4605bb3d807b86027680bc49e7fc40743c28cd3fdd03302a2e80ab2101794de4 not found: ID does not exist" containerID="4605bb3d807b86027680bc49e7fc40743c28cd3fdd03302a2e80ab2101794de4"
May 20 14:14:17.325965 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.325934    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4605bb3d807b86027680bc49e7fc40743c28cd3fdd03302a2e80ab2101794de4"} err="failed to get container status \"4605bb3d807b86027680bc49e7fc40743c28cd3fdd03302a2e80ab2101794de4\": rpc error: code = NotFound desc = could not find container \"4605bb3d807b86027680bc49e7fc40743c28cd3fdd03302a2e80ab2101794de4\": container with ID starting with 4605bb3d807b86027680bc49e7fc40743c28cd3fdd03302a2e80ab2101794de4 not found: ID does not exist"
May 20 14:14:17.329395 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.329368    2570 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-14a90-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/677266f2-8375-4dcf-a101-0160c6f8f5b4-error-404-isvc-14a90-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:14:17.329567 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.329401    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqwp7\" (UniqueName: \"kubernetes.io/projected/677266f2-8375-4dcf-a101-0160c6f8f5b4-kube-api-access-lqwp7\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:14:17.329567 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.329416    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/677266f2-8375-4dcf-a101-0160c6f8f5b4-proxy-tls\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:14:17.331414 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.331386    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"]
May 20 14:14:17.336135 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.336114    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg"]
May 20 14:14:17.364679 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:17.364641    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="677266f2-8375-4dcf-a101-0160c6f8f5b4" path="/var/lib/kubelet/pods/677266f2-8375-4dcf-a101-0160c6f8f5b4/volumes"
May 20 14:14:18.040323 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:18.040273    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-14a90-predictor-547774646f-lrxtg" podUID="677266f2-8375-4dcf-a101-0160c6f8f5b4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
May 20 14:14:21.261458 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:21.261378    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf" podUID="faae5ee3-1a59-4460-86dd-a79d893701c1" containerName="sequence-graph-14a90" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:14:22.193735 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:22.193690    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4" podUID="c82902d1-af79-484e-a67e-1f0ed4b08fc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 20 14:14:22.314324 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:22.314295    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"
May 20 14:14:22.314828 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:22.314803    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w" podUID="7bb62959-33c7-4cf3-824a-616ee8f239e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
May 20 14:14:26.261052 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:26.261008    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf" podUID="faae5ee3-1a59-4460-86dd-a79d893701c1" containerName="sequence-graph-14a90" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:14:26.261438 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:26.261126    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf"
May 20 14:14:31.260413 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:31.260375    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf" podUID="faae5ee3-1a59-4460-86dd-a79d893701c1" containerName="sequence-graph-14a90" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:14:32.194687 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:32.194656    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"
May 20 14:14:32.315106 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:32.315063    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w" podUID="7bb62959-33c7-4cf3-824a-616ee8f239e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
May 20 14:14:36.260681 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:36.260639    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf" podUID="faae5ee3-1a59-4460-86dd-a79d893701c1" containerName="sequence-graph-14a90" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:14:41.260591 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:41.260550    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf" podUID="faae5ee3-1a59-4460-86dd-a79d893701c1" containerName="sequence-graph-14a90" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:14:42.315421 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:42.315378    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w" podUID="7bb62959-33c7-4cf3-824a-616ee8f239e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
May 20 14:14:43.550739 ip-10-0-136-75 kubenswrapper[2570]: W0520 14:14:43.550697    2570 helpers.go:245] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaae5ee3_1a59_4460_86dd_a79d893701c1.slice/crio-conmon-d86c55d24646324575cae8909d828eab462a21e647e2bbb24443e68a18322b24.scope/memory.swap.max": read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaae5ee3_1a59_4460_86dd_a79d893701c1.slice/crio-conmon-d86c55d24646324575cae8909d828eab462a21e647e2bbb24443e68a18322b24.scope/memory.swap.max: no such device
May 20 14:14:43.705337 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:43.705308    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf"
May 20 14:14:43.746679 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:43.746638    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/faae5ee3-1a59-4460-86dd-a79d893701c1-openshift-service-ca-bundle\") pod \"faae5ee3-1a59-4460-86dd-a79d893701c1\" (UID: \"faae5ee3-1a59-4460-86dd-a79d893701c1\") "
May 20 14:14:43.746876 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:43.746730    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/faae5ee3-1a59-4460-86dd-a79d893701c1-proxy-tls\") pod \"faae5ee3-1a59-4460-86dd-a79d893701c1\" (UID: \"faae5ee3-1a59-4460-86dd-a79d893701c1\") "
May 20 14:14:43.747011 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:43.746984    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faae5ee3-1a59-4460-86dd-a79d893701c1-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "faae5ee3-1a59-4460-86dd-a79d893701c1" (UID: "faae5ee3-1a59-4460-86dd-a79d893701c1"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:14:43.749040 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:43.749017    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faae5ee3-1a59-4460-86dd-a79d893701c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "faae5ee3-1a59-4460-86dd-a79d893701c1" (UID: "faae5ee3-1a59-4460-86dd-a79d893701c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:14:43.847538 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:43.847490    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/faae5ee3-1a59-4460-86dd-a79d893701c1-openshift-service-ca-bundle\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:14:43.847538 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:43.847523    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/faae5ee3-1a59-4460-86dd-a79d893701c1-proxy-tls\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:14:44.007465 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.007382    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq"]
May 20 14:14:44.007808 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.007793    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="677266f2-8375-4dcf-a101-0160c6f8f5b4" containerName="kserve-container"
May 20 14:14:44.007866 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.007810    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="677266f2-8375-4dcf-a101-0160c6f8f5b4" containerName="kserve-container"
May 20 14:14:44.007866 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.007820    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faae5ee3-1a59-4460-86dd-a79d893701c1" containerName="sequence-graph-14a90"
May 20 14:14:44.007866 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.007825    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="faae5ee3-1a59-4460-86dd-a79d893701c1" containerName="sequence-graph-14a90"
May 20 14:14:44.007866 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.007848    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="677266f2-8375-4dcf-a101-0160c6f8f5b4" containerName="kube-rbac-proxy"
May 20 14:14:44.007866 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.007853    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="677266f2-8375-4dcf-a101-0160c6f8f5b4" containerName="kube-rbac-proxy"
May 20 14:14:44.007866 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.007861    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6304b107-7353-4d51-a2c4-7880f1998511" containerName="ensemble-graph-15cdb"
May 20 14:14:44.007866 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.007866    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6304b107-7353-4d51-a2c4-7880f1998511" containerName="ensemble-graph-15cdb"
May 20 14:14:44.008116 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.007919    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="faae5ee3-1a59-4460-86dd-a79d893701c1" containerName="sequence-graph-14a90"
May 20 14:14:44.008116 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.007927    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="677266f2-8375-4dcf-a101-0160c6f8f5b4" containerName="kube-rbac-proxy"
May 20 14:14:44.008116 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.007935    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="6304b107-7353-4d51-a2c4-7880f1998511" containerName="ensemble-graph-15cdb"
May 20 14:14:44.008116 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.007942    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="677266f2-8375-4dcf-a101-0160c6f8f5b4" containerName="kserve-container"
May 20 14:14:44.011087 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.011066    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq"
May 20 14:14:44.014010 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.013989    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-ecafe-serving-cert\""
May 20 14:14:44.014153 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.014052    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-ecafe-kube-rbac-proxy-sar-config\""
May 20 14:14:44.022897 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.022870    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq"]
May 20 14:14:44.049728 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.049695    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d-openshift-service-ca-bundle\") pod \"ensemble-graph-ecafe-fd8585c74-tmpzq\" (UID: \"ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d\") " pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq"
May 20 14:14:44.049901 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.049778    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d-proxy-tls\") pod \"ensemble-graph-ecafe-fd8585c74-tmpzq\" (UID: \"ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d\") " pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq"
May 20 14:14:44.151010 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.150969    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d-openshift-service-ca-bundle\") pod \"ensemble-graph-ecafe-fd8585c74-tmpzq\" (UID: \"ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d\") " pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq"
May 20 14:14:44.151215 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.151189    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d-proxy-tls\") pod \"ensemble-graph-ecafe-fd8585c74-tmpzq\" (UID: \"ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d\") " pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq"
May 20 14:14:44.151379 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:14:44.151357    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-ecafe-serving-cert: secret "ensemble-graph-ecafe-serving-cert" not found
May 20 14:14:44.151446 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:14:44.151435    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d-proxy-tls podName:ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d nodeName:}" failed. No retries permitted until 2026-05-20 14:14:44.651410976 +0000 UTC m=+1451.889027602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d-proxy-tls") pod "ensemble-graph-ecafe-fd8585c74-tmpzq" (UID: "ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d") : secret "ensemble-graph-ecafe-serving-cert" not found
May 20 14:14:44.151798 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.151782    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d-openshift-service-ca-bundle\") pod \"ensemble-graph-ecafe-fd8585c74-tmpzq\" (UID: \"ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d\") " pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq"
May 20 14:14:44.402628 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.402582    2570 generic.go:358] "Generic (PLEG): container finished" podID="faae5ee3-1a59-4460-86dd-a79d893701c1" containerID="d86c55d24646324575cae8909d828eab462a21e647e2bbb24443e68a18322b24" exitCode=0
May 20 14:14:44.402822 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.402645    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf" event={"ID":"faae5ee3-1a59-4460-86dd-a79d893701c1","Type":"ContainerDied","Data":"d86c55d24646324575cae8909d828eab462a21e647e2bbb24443e68a18322b24"}
May 20 14:14:44.402822 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.402681    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf" event={"ID":"faae5ee3-1a59-4460-86dd-a79d893701c1","Type":"ContainerDied","Data":"aaf9b56d41c50d3afe4637d333a9b7247f13d7ab22b830815dcf82312c82c344"}
May 20 14:14:44.402822 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.402702    2570 scope.go:117] "RemoveContainer" containerID="d86c55d24646324575cae8909d828eab462a21e647e2bbb24443e68a18322b24"
May 20 14:14:44.402822 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.402657    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf"
May 20 14:14:44.411568 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.411550    2570 scope.go:117] "RemoveContainer" containerID="d86c55d24646324575cae8909d828eab462a21e647e2bbb24443e68a18322b24"
May 20 14:14:44.411863 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:14:44.411845    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d86c55d24646324575cae8909d828eab462a21e647e2bbb24443e68a18322b24\": container with ID starting with d86c55d24646324575cae8909d828eab462a21e647e2bbb24443e68a18322b24 not found: ID does not exist" containerID="d86c55d24646324575cae8909d828eab462a21e647e2bbb24443e68a18322b24"
May 20 14:14:44.411924 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.411871    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d86c55d24646324575cae8909d828eab462a21e647e2bbb24443e68a18322b24"} err="failed to get container status \"d86c55d24646324575cae8909d828eab462a21e647e2bbb24443e68a18322b24\": rpc error: code = NotFound desc = could not find container \"d86c55d24646324575cae8909d828eab462a21e647e2bbb24443e68a18322b24\": container with ID starting with d86c55d24646324575cae8909d828eab462a21e647e2bbb24443e68a18322b24 not found: ID does not exist"
May 20 14:14:44.426827 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.426795    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf"]
May 20 14:14:44.431348 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.431321    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-14a90-6db7cfb9f9-9d8xf"]
May 20 14:14:44.656214 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.656120    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d-proxy-tls\") pod \"ensemble-graph-ecafe-fd8585c74-tmpzq\" (UID: \"ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d\") " pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq"
May 20 14:14:44.658805 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.658779    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d-proxy-tls\") pod \"ensemble-graph-ecafe-fd8585c74-tmpzq\" (UID: \"ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d\") " pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq"
May 20 14:14:44.922472 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:44.922391    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq"
May 20 14:14:45.076794 ip-10-0-136-75 kubenswrapper[2570]: W0520 14:14:45.076748    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa3794a_a4aa_43fe_a2f5_e6d3a576ee5d.slice/crio-6cd6bf2f0cb3ee901012e81ebd182f894f582d414b8f39ec9fb7fed32b15485e WatchSource:0}: Error finding container 6cd6bf2f0cb3ee901012e81ebd182f894f582d414b8f39ec9fb7fed32b15485e: Status 404 returned error can't find the container with id 6cd6bf2f0cb3ee901012e81ebd182f894f582d414b8f39ec9fb7fed32b15485e
May 20 14:14:45.076794 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:45.076775    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq"]
May 20 14:14:45.363720 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:45.363676    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faae5ee3-1a59-4460-86dd-a79d893701c1" path="/var/lib/kubelet/pods/faae5ee3-1a59-4460-86dd-a79d893701c1/volumes"
May 20 14:14:45.407279 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:45.407245    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq" event={"ID":"ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d","Type":"ContainerStarted","Data":"6fea0bee5e3009a8f71a7cf1d7c7abd8cb5b3f28841f4abf17f8b5f7b820c029"}
May 20 14:14:45.407279 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:45.407284    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq" event={"ID":"ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d","Type":"ContainerStarted","Data":"6cd6bf2f0cb3ee901012e81ebd182f894f582d414b8f39ec9fb7fed32b15485e"}
May 20 14:14:45.407545 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:45.407361    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq"
May 20 14:14:45.428249 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:45.428192    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq" podStartSLOduration=2.428176925 podStartE2EDuration="2.428176925s" podCreationTimestamp="2026-05-20 14:14:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:14:45.425661123 +0000 UTC m=+1452.663277771" watchObservedRunningTime="2026-05-20 14:14:45.428176925 +0000 UTC m=+1452.665793572"
May 20 14:14:51.417554 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:51.417523    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq"
May 20 14:14:52.315318 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:14:52.315273    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w" podUID="7bb62959-33c7-4cf3-824a-616ee8f239e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
May 20 14:15:02.315815 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:15:02.315774    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"
May 20 14:15:13.829731 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:15:13.829686    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt"]
May 20 14:15:13.833225 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:15:13.833208    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt"
May 20 14:15:13.835928 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:15:13.835901    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-734fc-serving-cert\""
May 20 14:15:13.836066 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:15:13.835903    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-734fc-kube-rbac-proxy-sar-config\""
May 20 14:15:13.844288 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:15:13.844261    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt"]
May 20 14:15:13.911981 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:15:13.911946    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/881877ca-fd37-439c-b536-73be79de984c-proxy-tls\") pod \"sequence-graph-734fc-699db48bbd-8gvbt\" (UID: \"881877ca-fd37-439c-b536-73be79de984c\") " pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt"
May 20 14:15:13.912180 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:15:13.912004    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/881877ca-fd37-439c-b536-73be79de984c-openshift-service-ca-bundle\") pod \"sequence-graph-734fc-699db48bbd-8gvbt\" (UID: \"881877ca-fd37-439c-b536-73be79de984c\") " pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt"
May 20 14:15:14.012983 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:15:14.012944    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/881877ca-fd37-439c-b536-73be79de984c-openshift-service-ca-bundle\") pod \"sequence-graph-734fc-699db48bbd-8gvbt\" (UID: \"881877ca-fd37-439c-b536-73be79de984c\") " pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt"
May 20 14:15:14.013187 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:15:14.013046    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/881877ca-fd37-439c-b536-73be79de984c-proxy-tls\") pod \"sequence-graph-734fc-699db48bbd-8gvbt\" (UID: \"881877ca-fd37-439c-b536-73be79de984c\") " pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt"
May 20 14:15:14.013187 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:15:14.013156    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-734fc-serving-cert: secret "sequence-graph-734fc-serving-cert" not found
May 20 14:15:14.013312 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:15:14.013217    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/881877ca-fd37-439c-b536-73be79de984c-proxy-tls podName:881877ca-fd37-439c-b536-73be79de984c nodeName:}" failed. No retries permitted until 2026-05-20 14:15:14.5131996 +0000 UTC m=+1481.750816225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/881877ca-fd37-439c-b536-73be79de984c-proxy-tls") pod "sequence-graph-734fc-699db48bbd-8gvbt" (UID: "881877ca-fd37-439c-b536-73be79de984c") : secret "sequence-graph-734fc-serving-cert" not found
May 20 14:15:14.013691 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:15:14.013665    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/881877ca-fd37-439c-b536-73be79de984c-openshift-service-ca-bundle\") pod \"sequence-graph-734fc-699db48bbd-8gvbt\" (UID: \"881877ca-fd37-439c-b536-73be79de984c\") " pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt"
May 20 14:15:14.517053 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:15:14.517012    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/881877ca-fd37-439c-b536-73be79de984c-proxy-tls\") pod \"sequence-graph-734fc-699db48bbd-8gvbt\" (UID: \"881877ca-fd37-439c-b536-73be79de984c\") " pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt"
May 20 14:15:14.519716 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:15:14.519694    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/881877ca-fd37-439c-b536-73be79de984c-proxy-tls\") pod \"sequence-graph-734fc-699db48bbd-8gvbt\" (UID: \"881877ca-fd37-439c-b536-73be79de984c\") " pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt"
May 20 14:15:14.745607 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:15:14.745552    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt"
May 20 14:15:14.883009 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:15:14.882980    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt"]
May 20 14:15:14.885133 ip-10-0-136-75 kubenswrapper[2570]: W0520 14:15:14.885104    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod881877ca_fd37_439c_b536_73be79de984c.slice/crio-c26d2ef28cdb5ebe2f116a48cb26139d2030e9a5fb8c49921290089a231ff246 WatchSource:0}: Error finding container c26d2ef28cdb5ebe2f116a48cb26139d2030e9a5fb8c49921290089a231ff246: Status 404 returned error can't find the container with id c26d2ef28cdb5ebe2f116a48cb26139d2030e9a5fb8c49921290089a231ff246
May 20 14:15:15.507053 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:15:15.507019    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt" event={"ID":"881877ca-fd37-439c-b536-73be79de984c","Type":"ContainerStarted","Data":"1c1f0131c1b7c9d67bdb14fde0ca971c1e77bb4cad94b69cdb159c5f9e76a6cc"}
May 20 14:15:15.507234 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:15:15.507061    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt" event={"ID":"881877ca-fd37-439c-b536-73be79de984c","Type":"ContainerStarted","Data":"c26d2ef28cdb5ebe2f116a48cb26139d2030e9a5fb8c49921290089a231ff246"}
May 20 14:15:15.507234 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:15:15.507135    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt"
May 20 14:15:15.526017 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:15:15.525966    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt" podStartSLOduration=2.525950821 podStartE2EDuration="2.525950821s" podCreationTimestamp="2026-05-20 14:15:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:15:15.52476379 +0000 UTC m=+1482.762380438" watchObservedRunningTime="2026-05-20 14:15:15.525950821 +0000 UTC m=+1482.763567467"
May 20 14:15:21.517220 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:15:21.517187    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt"
May 20 14:15:33.352073 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:15:33.352046    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 14:15:33.355677 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:15:33.355643    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 14:20:33.377568 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:20:33.377538    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 14:20:33.381650 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:20:33.381625    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 14:22:58.598698 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:58.598662    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq"]
May 20 14:22:58.601187 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:58.598870    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq" podUID="ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d" containerName="ensemble-graph-ecafe" containerID="cri-o://6fea0bee5e3009a8f71a7cf1d7c7abd8cb5b3f28841f4abf17f8b5f7b820c029" gracePeriod=30
May 20 14:22:58.846797 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:58.846759    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"]
May 20 14:22:58.847379 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:58.847319    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4" podUID="c82902d1-af79-484e-a67e-1f0ed4b08fc9" containerName="kserve-container" containerID="cri-o://a0b7ea4d2c83a0f41e65d7a252717350071d2c8f227a55ba36c56e54ce01179a" gracePeriod=30
May 20 14:22:58.847530 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:58.847327    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4" podUID="c82902d1-af79-484e-a67e-1f0ed4b08fc9" containerName="kube-rbac-proxy" containerID="cri-o://262fc6817d94dda942fd48375e3c90fe6c28f0127a7b112216983ea622a2484a" gracePeriod=30
May 20 14:22:58.906549 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:58.906514    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"]
May 20 14:22:58.909977 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:58.909959    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"
May 20 14:22:58.912526 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:58.912501    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a868a-predictor-serving-cert\""
May 20 14:22:58.912888 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:58.912869    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a868a-kube-rbac-proxy-sar-config\""
May 20 14:22:58.923572 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:58.923532    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"]
May 20 14:22:59.038788 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:59.038754    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj78s\" (UniqueName: \"kubernetes.io/projected/b151f8aa-7bf0-4a45-8612-a746025bcae7-kube-api-access-nj78s\") pod \"error-404-isvc-a868a-predictor-848b5648c5-46cjq\" (UID: \"b151f8aa-7bf0-4a45-8612-a746025bcae7\") " pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"
May 20 14:22:59.038788 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:59.038798    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b151f8aa-7bf0-4a45-8612-a746025bcae7-proxy-tls\") pod \"error-404-isvc-a868a-predictor-848b5648c5-46cjq\" (UID: \"b151f8aa-7bf0-4a45-8612-a746025bcae7\") " pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"
May 20 14:22:59.039046 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:59.038895    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-a868a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b151f8aa-7bf0-4a45-8612-a746025bcae7-error-404-isvc-a868a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a868a-predictor-848b5648c5-46cjq\" (UID: \"b151f8aa-7bf0-4a45-8612-a746025bcae7\") " pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"
May 20 14:22:59.105124 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:59.105035    2570 generic.go:358] "Generic (PLEG): container finished" podID="c82902d1-af79-484e-a67e-1f0ed4b08fc9" containerID="262fc6817d94dda942fd48375e3c90fe6c28f0127a7b112216983ea622a2484a" exitCode=2
May 20 14:22:59.105124 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:59.105105    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4" event={"ID":"c82902d1-af79-484e-a67e-1f0ed4b08fc9","Type":"ContainerDied","Data":"262fc6817d94dda942fd48375e3c90fe6c28f0127a7b112216983ea622a2484a"}
May 20 14:22:59.139810 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:59.139768    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nj78s\" (UniqueName: \"kubernetes.io/projected/b151f8aa-7bf0-4a45-8612-a746025bcae7-kube-api-access-nj78s\") pod \"error-404-isvc-a868a-predictor-848b5648c5-46cjq\" (UID: \"b151f8aa-7bf0-4a45-8612-a746025bcae7\") " pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"
May 20 14:22:59.139810 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:59.139813    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b151f8aa-7bf0-4a45-8612-a746025bcae7-proxy-tls\") pod \"error-404-isvc-a868a-predictor-848b5648c5-46cjq\" (UID: \"b151f8aa-7bf0-4a45-8612-a746025bcae7\") " pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"
May 20 14:22:59.140063 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:59.139879    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-a868a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b151f8aa-7bf0-4a45-8612-a746025bcae7-error-404-isvc-a868a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a868a-predictor-848b5648c5-46cjq\" (UID: \"b151f8aa-7bf0-4a45-8612-a746025bcae7\") " pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"
May 20 14:22:59.140063 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:22:59.139986    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-a868a-predictor-serving-cert: secret "error-404-isvc-a868a-predictor-serving-cert" not found
May 20 14:22:59.140173 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:22:59.140065    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b151f8aa-7bf0-4a45-8612-a746025bcae7-proxy-tls podName:b151f8aa-7bf0-4a45-8612-a746025bcae7 nodeName:}" failed. No retries permitted until 2026-05-20 14:22:59.640047688 +0000 UTC m=+1946.877664314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b151f8aa-7bf0-4a45-8612-a746025bcae7-proxy-tls") pod "error-404-isvc-a868a-predictor-848b5648c5-46cjq" (UID: "b151f8aa-7bf0-4a45-8612-a746025bcae7") : secret "error-404-isvc-a868a-predictor-serving-cert" not found
May 20 14:22:59.140512 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:59.140493    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-a868a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b151f8aa-7bf0-4a45-8612-a746025bcae7-error-404-isvc-a868a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a868a-predictor-848b5648c5-46cjq\" (UID: \"b151f8aa-7bf0-4a45-8612-a746025bcae7\") " pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"
May 20 14:22:59.150110 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:59.150060    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj78s\" (UniqueName: \"kubernetes.io/projected/b151f8aa-7bf0-4a45-8612-a746025bcae7-kube-api-access-nj78s\") pod \"error-404-isvc-a868a-predictor-848b5648c5-46cjq\" (UID: \"b151f8aa-7bf0-4a45-8612-a746025bcae7\") " pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"
May 20 14:22:59.644864 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:59.644819    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b151f8aa-7bf0-4a45-8612-a746025bcae7-proxy-tls\") pod \"error-404-isvc-a868a-predictor-848b5648c5-46cjq\" (UID: \"b151f8aa-7bf0-4a45-8612-a746025bcae7\") " pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"
May 20 14:22:59.647361 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:59.647339    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b151f8aa-7bf0-4a45-8612-a746025bcae7-proxy-tls\") pod \"error-404-isvc-a868a-predictor-848b5648c5-46cjq\" (UID: \"b151f8aa-7bf0-4a45-8612-a746025bcae7\") " pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"
May 20 14:22:59.820867 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:59.820831    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"
May 20 14:22:59.963636 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:59.963592    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"]
May 20 14:22:59.966269 ip-10-0-136-75 kubenswrapper[2570]: W0520 14:22:59.966240    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb151f8aa_7bf0_4a45_8612_a746025bcae7.slice/crio-35dc3ffc9650c41db51955d0e1623382c9a60fe5e3733db28dfdf50fd6ed0ec4 WatchSource:0}: Error finding container 35dc3ffc9650c41db51955d0e1623382c9a60fe5e3733db28dfdf50fd6ed0ec4: Status 404 returned error can't find the container with id 35dc3ffc9650c41db51955d0e1623382c9a60fe5e3733db28dfdf50fd6ed0ec4
May 20 14:22:59.968143 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:22:59.968122    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:23:00.110169 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:00.110135    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq" event={"ID":"b151f8aa-7bf0-4a45-8612-a746025bcae7","Type":"ContainerStarted","Data":"b1ae5c6e014f8b2d00d6ad72d7001ec698fa0d49da408283e5316231b3d45dda"}
May 20 14:23:00.110169 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:00.110175    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq" event={"ID":"b151f8aa-7bf0-4a45-8612-a746025bcae7","Type":"ContainerStarted","Data":"805f9854eaa2ce27ee93211de65a00399bca7f465b26aad765be6dda1f399073"}
May 20 14:23:00.110341 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:00.110184    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq" event={"ID":"b151f8aa-7bf0-4a45-8612-a746025bcae7","Type":"ContainerStarted","Data":"35dc3ffc9650c41db51955d0e1623382c9a60fe5e3733db28dfdf50fd6ed0ec4"}
May 20 14:23:00.110341 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:00.110248    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"
May 20 14:23:00.130252 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:00.130191    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq" podStartSLOduration=2.130172259 podStartE2EDuration="2.130172259s" podCreationTimestamp="2026-05-20 14:22:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:23:00.130116816 +0000 UTC m=+1947.367733474" watchObservedRunningTime="2026-05-20 14:23:00.130172259 +0000 UTC m=+1947.367788903"
May 20 14:23:01.113480 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:01.113446    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"
May 20 14:23:01.114865 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:01.114827    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq" podUID="b151f8aa-7bf0-4a45-8612-a746025bcae7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 20 14:23:01.415728 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:01.415632    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq" podUID="ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d" containerName="ensemble-graph-ecafe" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:23:02.117041 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:02.116999    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq" podUID="b151f8aa-7bf0-4a45-8612-a746025bcae7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 20 14:23:02.189821 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:02.189763    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4" podUID="c82902d1-af79-484e-a67e-1f0ed4b08fc9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
May 20 14:23:02.194166 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:02.194138    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4" podUID="c82902d1-af79-484e-a67e-1f0ed4b08fc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 20 14:23:02.297843 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:02.297815    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"
May 20 14:23:02.370553 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:02.370455    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xt84\" (UniqueName: \"kubernetes.io/projected/c82902d1-af79-484e-a67e-1f0ed4b08fc9-kube-api-access-4xt84\") pod \"c82902d1-af79-484e-a67e-1f0ed4b08fc9\" (UID: \"c82902d1-af79-484e-a67e-1f0ed4b08fc9\") "
May 20 14:23:02.370553 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:02.370532    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c82902d1-af79-484e-a67e-1f0ed4b08fc9-proxy-tls\") pod \"c82902d1-af79-484e-a67e-1f0ed4b08fc9\" (UID: \"c82902d1-af79-484e-a67e-1f0ed4b08fc9\") "
May 20 14:23:02.370830 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:02.370575    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-ecafe-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c82902d1-af79-484e-a67e-1f0ed4b08fc9-error-404-isvc-ecafe-kube-rbac-proxy-sar-config\") pod \"c82902d1-af79-484e-a67e-1f0ed4b08fc9\" (UID: \"c82902d1-af79-484e-a67e-1f0ed4b08fc9\") "
May 20 14:23:02.370992 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:02.370966    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82902d1-af79-484e-a67e-1f0ed4b08fc9-error-404-isvc-ecafe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-ecafe-kube-rbac-proxy-sar-config") pod "c82902d1-af79-484e-a67e-1f0ed4b08fc9" (UID: "c82902d1-af79-484e-a67e-1f0ed4b08fc9"). InnerVolumeSpecName "error-404-isvc-ecafe-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:23:02.372871 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:02.372838    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c82902d1-af79-484e-a67e-1f0ed4b08fc9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c82902d1-af79-484e-a67e-1f0ed4b08fc9" (UID: "c82902d1-af79-484e-a67e-1f0ed4b08fc9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:23:02.372976 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:02.372956    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82902d1-af79-484e-a67e-1f0ed4b08fc9-kube-api-access-4xt84" (OuterVolumeSpecName: "kube-api-access-4xt84") pod "c82902d1-af79-484e-a67e-1f0ed4b08fc9" (UID: "c82902d1-af79-484e-a67e-1f0ed4b08fc9"). InnerVolumeSpecName "kube-api-access-4xt84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:23:02.471568 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:02.471512    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xt84\" (UniqueName: \"kubernetes.io/projected/c82902d1-af79-484e-a67e-1f0ed4b08fc9-kube-api-access-4xt84\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:23:02.471568 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:02.471565    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c82902d1-af79-484e-a67e-1f0ed4b08fc9-proxy-tls\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:23:02.471793 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:02.471584    2570 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-ecafe-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c82902d1-af79-484e-a67e-1f0ed4b08fc9-error-404-isvc-ecafe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:23:03.121890 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:03.121853    2570 generic.go:358] "Generic (PLEG): container finished" podID="c82902d1-af79-484e-a67e-1f0ed4b08fc9" containerID="a0b7ea4d2c83a0f41e65d7a252717350071d2c8f227a55ba36c56e54ce01179a" exitCode=0
May 20 14:23:03.122389 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:03.121941    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"
May 20 14:23:03.122389 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:03.121944    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4" event={"ID":"c82902d1-af79-484e-a67e-1f0ed4b08fc9","Type":"ContainerDied","Data":"a0b7ea4d2c83a0f41e65d7a252717350071d2c8f227a55ba36c56e54ce01179a"}
May 20 14:23:03.122389 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:03.121983    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4" event={"ID":"c82902d1-af79-484e-a67e-1f0ed4b08fc9","Type":"ContainerDied","Data":"a5a9b4512bd96503e8fd4dd4acdc6ff4cf3d1e977441b46ed2c06953b207889e"}
May 20 14:23:03.122389 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:03.121999    2570 scope.go:117] "RemoveContainer" containerID="262fc6817d94dda942fd48375e3c90fe6c28f0127a7b112216983ea622a2484a"
May 20 14:23:03.131976 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:03.131579    2570 scope.go:117] "RemoveContainer" containerID="a0b7ea4d2c83a0f41e65d7a252717350071d2c8f227a55ba36c56e54ce01179a"
May 20 14:23:03.141438 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:03.141420    2570 scope.go:117] "RemoveContainer" containerID="262fc6817d94dda942fd48375e3c90fe6c28f0127a7b112216983ea622a2484a"
May 20 14:23:03.141759 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:23:03.141739    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"262fc6817d94dda942fd48375e3c90fe6c28f0127a7b112216983ea622a2484a\": container with ID starting with 262fc6817d94dda942fd48375e3c90fe6c28f0127a7b112216983ea622a2484a not found: ID does not exist" containerID="262fc6817d94dda942fd48375e3c90fe6c28f0127a7b112216983ea622a2484a"
May 20 14:23:03.141825 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:03.141770    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"262fc6817d94dda942fd48375e3c90fe6c28f0127a7b112216983ea622a2484a"} err="failed to get container status \"262fc6817d94dda942fd48375e3c90fe6c28f0127a7b112216983ea622a2484a\": rpc error: code = NotFound desc = could not find container \"262fc6817d94dda942fd48375e3c90fe6c28f0127a7b112216983ea622a2484a\": container with ID starting with 262fc6817d94dda942fd48375e3c90fe6c28f0127a7b112216983ea622a2484a not found: ID does not exist"
May 20 14:23:03.141825 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:03.141789    2570 scope.go:117] "RemoveContainer" containerID="a0b7ea4d2c83a0f41e65d7a252717350071d2c8f227a55ba36c56e54ce01179a"
May 20 14:23:03.142070 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:23:03.142044    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b7ea4d2c83a0f41e65d7a252717350071d2c8f227a55ba36c56e54ce01179a\": container with ID starting with a0b7ea4d2c83a0f41e65d7a252717350071d2c8f227a55ba36c56e54ce01179a not found: ID does not exist" containerID="a0b7ea4d2c83a0f41e65d7a252717350071d2c8f227a55ba36c56e54ce01179a"
May 20 14:23:03.142122 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:03.142086    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b7ea4d2c83a0f41e65d7a252717350071d2c8f227a55ba36c56e54ce01179a"} err="failed to get container status \"a0b7ea4d2c83a0f41e65d7a252717350071d2c8f227a55ba36c56e54ce01179a\": rpc error: code = NotFound desc = could not find container \"a0b7ea4d2c83a0f41e65d7a252717350071d2c8f227a55ba36c56e54ce01179a\": container with ID starting with a0b7ea4d2c83a0f41e65d7a252717350071d2c8f227a55ba36c56e54ce01179a not found: ID does not exist"
May 20 14:23:03.148933 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:03.148903    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"]
May 20 14:23:03.152425 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:03.152398    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ecafe-predictor-bf7d7ddc-5k2k4"]
May 20 14:23:03.166118 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:23:03.166087    2570 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82902d1_af79_484e_a67e_1f0ed4b08fc9.slice/crio-a5a9b4512bd96503e8fd4dd4acdc6ff4cf3d1e977441b46ed2c06953b207889e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82902d1_af79_484e_a67e_1f0ed4b08fc9.slice\": RecentStats: unable to find data in memory cache]"
May 20 14:23:03.365288 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:03.365251    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c82902d1-af79-484e-a67e-1f0ed4b08fc9" path="/var/lib/kubelet/pods/c82902d1-af79-484e-a67e-1f0ed4b08fc9/volumes"
May 20 14:23:06.415493 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:06.415451    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq" podUID="ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d" containerName="ensemble-graph-ecafe" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:23:07.121654 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:07.121605    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"
May 20 14:23:07.122166 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:07.122137    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq" podUID="b151f8aa-7bf0-4a45-8612-a746025bcae7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 20 14:23:11.414992 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:11.414953    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq" podUID="ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d" containerName="ensemble-graph-ecafe" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:23:11.415374 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:11.415056    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq"
May 20 14:23:16.415187 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:16.415146    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq" podUID="ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d" containerName="ensemble-graph-ecafe" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:23:17.122507 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:17.122466    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq" podUID="b151f8aa-7bf0-4a45-8612-a746025bcae7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 20 14:23:21.415373 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:21.415284    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq" podUID="ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d" containerName="ensemble-graph-ecafe" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:23:26.415396 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:26.415360    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq" podUID="ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d" containerName="ensemble-graph-ecafe" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:23:27.123061 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:27.123016    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq" podUID="b151f8aa-7bf0-4a45-8612-a746025bcae7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 20 14:23:28.409759 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.409697    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt"]
May 20 14:23:28.410176 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.409948    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt" podUID="881877ca-fd37-439c-b536-73be79de984c" containerName="sequence-graph-734fc" containerID="cri-o://1c1f0131c1b7c9d67bdb14fde0ca971c1e77bb4cad94b69cdb159c5f9e76a6cc" gracePeriod=30
May 20 14:23:28.512776 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.512744    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"]
May 20 14:23:28.513590 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.513552    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w" podUID="7bb62959-33c7-4cf3-824a-616ee8f239e2" containerName="kserve-container" containerID="cri-o://3b077e31529d7078dcc0f4d7557b123050863156e7d01674a9eab49c28a7ff1e" gracePeriod=30
May 20 14:23:28.513829 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.513776    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w" podUID="7bb62959-33c7-4cf3-824a-616ee8f239e2" containerName="kube-rbac-proxy" containerID="cri-o://1d4eb9d5e21f33468d91cbedb36c06b2fe1afd09615cbe34382c4790a83e9b34" gracePeriod=30
May 20 14:23:28.777798 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.777772    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq"
May 20 14:23:28.810239 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.810203    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d-proxy-tls\") pod \"ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d\" (UID: \"ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d\") "
May 20 14:23:28.810421 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.810301    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d-openshift-service-ca-bundle\") pod \"ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d\" (UID: \"ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d\") "
May 20 14:23:28.810753 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.810723    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d" (UID: "ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:23:28.812975 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.812941    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d" (UID: "ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:23:28.821963 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.821934    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"]
May 20 14:23:28.822572 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.822553    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d" containerName="ensemble-graph-ecafe"
May 20 14:23:28.822636 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.822578    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d" containerName="ensemble-graph-ecafe"
May 20 14:23:28.822636 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.822631    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c82902d1-af79-484e-a67e-1f0ed4b08fc9" containerName="kube-rbac-proxy"
May 20 14:23:28.822709 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.822642    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c82902d1-af79-484e-a67e-1f0ed4b08fc9" containerName="kube-rbac-proxy"
May 20 14:23:28.822709 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.822659    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c82902d1-af79-484e-a67e-1f0ed4b08fc9" containerName="kserve-container"
May 20 14:23:28.822709 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.822668    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c82902d1-af79-484e-a67e-1f0ed4b08fc9" containerName="kserve-container"
May 20 14:23:28.822805 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.822775    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d" containerName="ensemble-graph-ecafe"
May 20 14:23:28.822805 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.822790    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="c82902d1-af79-484e-a67e-1f0ed4b08fc9" containerName="kserve-container"
May 20 14:23:28.822866 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.822804    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="c82902d1-af79-484e-a67e-1f0ed4b08fc9" containerName="kube-rbac-proxy"
May 20 14:23:28.826819 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.826794    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"
May 20 14:23:28.829437 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.829411    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3e303-predictor-serving-cert\""
May 20 14:23:28.829530 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.829413    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3e303-kube-rbac-proxy-sar-config\""
May 20 14:23:28.837721 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.837693    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"]
May 20 14:23:28.911058 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.911004    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngb6q\" (UniqueName: \"kubernetes.io/projected/01a6cb98-37b5-49c2-a601-d09952a1df97-kube-api-access-ngb6q\") pod \"error-404-isvc-3e303-predictor-7ccc759659-gfmqm\" (UID: \"01a6cb98-37b5-49c2-a601-d09952a1df97\") " pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"
May 20 14:23:28.911241 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.911064    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-3e303-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01a6cb98-37b5-49c2-a601-d09952a1df97-error-404-isvc-3e303-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3e303-predictor-7ccc759659-gfmqm\" (UID: \"01a6cb98-37b5-49c2-a601-d09952a1df97\") " pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"
May 20 14:23:28.911241 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.911113    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01a6cb98-37b5-49c2-a601-d09952a1df97-proxy-tls\") pod \"error-404-isvc-3e303-predictor-7ccc759659-gfmqm\" (UID: \"01a6cb98-37b5-49c2-a601-d09952a1df97\") " pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"
May 20 14:23:28.911241 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.911216    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d-openshift-service-ca-bundle\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:23:28.911241 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:28.911239    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d-proxy-tls\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:23:29.012062 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:29.011954    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-3e303-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01a6cb98-37b5-49c2-a601-d09952a1df97-error-404-isvc-3e303-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3e303-predictor-7ccc759659-gfmqm\" (UID: \"01a6cb98-37b5-49c2-a601-d09952a1df97\") " pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"
May 20 14:23:29.012062 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:29.012009    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01a6cb98-37b5-49c2-a601-d09952a1df97-proxy-tls\") pod \"error-404-isvc-3e303-predictor-7ccc759659-gfmqm\" (UID: \"01a6cb98-37b5-49c2-a601-d09952a1df97\") " pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"
May 20 14:23:29.012311 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:29.012075    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ngb6q\" (UniqueName: \"kubernetes.io/projected/01a6cb98-37b5-49c2-a601-d09952a1df97-kube-api-access-ngb6q\") pod \"error-404-isvc-3e303-predictor-7ccc759659-gfmqm\" (UID: \"01a6cb98-37b5-49c2-a601-d09952a1df97\") " pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"
May 20 14:23:29.012311 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:23:29.012189    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-3e303-predictor-serving-cert: secret "error-404-isvc-3e303-predictor-serving-cert" not found
May 20 14:23:29.012311 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:23:29.012254    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01a6cb98-37b5-49c2-a601-d09952a1df97-proxy-tls podName:01a6cb98-37b5-49c2-a601-d09952a1df97 nodeName:}" failed. No retries permitted until 2026-05-20 14:23:29.512238094 +0000 UTC m=+1976.749854719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/01a6cb98-37b5-49c2-a601-d09952a1df97-proxy-tls") pod "error-404-isvc-3e303-predictor-7ccc759659-gfmqm" (UID: "01a6cb98-37b5-49c2-a601-d09952a1df97") : secret "error-404-isvc-3e303-predictor-serving-cert" not found
May 20 14:23:29.012710 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:29.012687    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-3e303-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01a6cb98-37b5-49c2-a601-d09952a1df97-error-404-isvc-3e303-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3e303-predictor-7ccc759659-gfmqm\" (UID: \"01a6cb98-37b5-49c2-a601-d09952a1df97\") " pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"
May 20 14:23:29.022392 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:29.022369    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngb6q\" (UniqueName: \"kubernetes.io/projected/01a6cb98-37b5-49c2-a601-d09952a1df97-kube-api-access-ngb6q\") pod \"error-404-isvc-3e303-predictor-7ccc759659-gfmqm\" (UID: \"01a6cb98-37b5-49c2-a601-d09952a1df97\") " pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"
May 20 14:23:29.219869 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:29.219837    2570 generic.go:358] "Generic (PLEG): container finished" podID="7bb62959-33c7-4cf3-824a-616ee8f239e2" containerID="1d4eb9d5e21f33468d91cbedb36c06b2fe1afd09615cbe34382c4790a83e9b34" exitCode=2
May 20 14:23:29.220041 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:29.219917    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w" event={"ID":"7bb62959-33c7-4cf3-824a-616ee8f239e2","Type":"ContainerDied","Data":"1d4eb9d5e21f33468d91cbedb36c06b2fe1afd09615cbe34382c4790a83e9b34"}
May 20 14:23:29.220891 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:29.220870    2570 generic.go:358] "Generic (PLEG): container finished" podID="ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d" containerID="6fea0bee5e3009a8f71a7cf1d7c7abd8cb5b3f28841f4abf17f8b5f7b820c029" exitCode=0
May 20 14:23:29.221001 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:29.220942    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq"
May 20 14:23:29.221001 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:29.220954    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq" event={"ID":"ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d","Type":"ContainerDied","Data":"6fea0bee5e3009a8f71a7cf1d7c7abd8cb5b3f28841f4abf17f8b5f7b820c029"}
May 20 14:23:29.221001 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:29.220989    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq" event={"ID":"ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d","Type":"ContainerDied","Data":"6cd6bf2f0cb3ee901012e81ebd182f894f582d414b8f39ec9fb7fed32b15485e"}
May 20 14:23:29.221106 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:29.221011    2570 scope.go:117] "RemoveContainer" containerID="6fea0bee5e3009a8f71a7cf1d7c7abd8cb5b3f28841f4abf17f8b5f7b820c029"
May 20 14:23:29.229372 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:29.229355    2570 scope.go:117] "RemoveContainer" containerID="6fea0bee5e3009a8f71a7cf1d7c7abd8cb5b3f28841f4abf17f8b5f7b820c029"
May 20 14:23:29.229643 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:23:29.229597    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fea0bee5e3009a8f71a7cf1d7c7abd8cb5b3f28841f4abf17f8b5f7b820c029\": container with ID starting with 6fea0bee5e3009a8f71a7cf1d7c7abd8cb5b3f28841f4abf17f8b5f7b820c029 not found: ID does not exist" containerID="6fea0bee5e3009a8f71a7cf1d7c7abd8cb5b3f28841f4abf17f8b5f7b820c029"
May 20 14:23:29.229742 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:29.229650    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fea0bee5e3009a8f71a7cf1d7c7abd8cb5b3f28841f4abf17f8b5f7b820c029"} err="failed to get container status \"6fea0bee5e3009a8f71a7cf1d7c7abd8cb5b3f28841f4abf17f8b5f7b820c029\": rpc error: code = NotFound desc = could not find container \"6fea0bee5e3009a8f71a7cf1d7c7abd8cb5b3f28841f4abf17f8b5f7b820c029\": container with ID starting with 6fea0bee5e3009a8f71a7cf1d7c7abd8cb5b3f28841f4abf17f8b5f7b820c029 not found: ID does not exist"
May 20 14:23:29.244324 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:29.244296    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq"]
May 20 14:23:29.249041 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:29.249012    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-ecafe-fd8585c74-tmpzq"]
May 20 14:23:29.364998 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:29.364961    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d" path="/var/lib/kubelet/pods/ffa3794a-a4aa-43fe-a2f5-e6d3a576ee5d/volumes"
May 20 14:23:29.516960 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:29.516929    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01a6cb98-37b5-49c2-a601-d09952a1df97-proxy-tls\") pod \"error-404-isvc-3e303-predictor-7ccc759659-gfmqm\" (UID: \"01a6cb98-37b5-49c2-a601-d09952a1df97\") " pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"
May 20 14:23:29.519606 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:29.519571    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01a6cb98-37b5-49c2-a601-d09952a1df97-proxy-tls\") pod \"error-404-isvc-3e303-predictor-7ccc759659-gfmqm\" (UID: \"01a6cb98-37b5-49c2-a601-d09952a1df97\") " pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"
May 20 14:23:29.738015 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:29.737918    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"
May 20 14:23:29.878683 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:29.878647    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"]
May 20 14:23:29.882093 ip-10-0-136-75 kubenswrapper[2570]: W0520 14:23:29.882061    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a6cb98_37b5_49c2_a601_d09952a1df97.slice/crio-f566fc3a260675ef776d4c070acc0d20550710359f4c1c1bbe9081c45e79d9eb WatchSource:0}: Error finding container f566fc3a260675ef776d4c070acc0d20550710359f4c1c1bbe9081c45e79d9eb: Status 404 returned error can't find the container with id f566fc3a260675ef776d4c070acc0d20550710359f4c1c1bbe9081c45e79d9eb
May 20 14:23:30.226709 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:30.226663    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm" event={"ID":"01a6cb98-37b5-49c2-a601-d09952a1df97","Type":"ContainerStarted","Data":"3380eb6d23c5226011e16a6450c34e764830590e714a6e091a4f434908d5ccd5"}
May 20 14:23:30.226709 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:30.226702    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm" event={"ID":"01a6cb98-37b5-49c2-a601-d09952a1df97","Type":"ContainerStarted","Data":"afcf040c1fc926c2cde9fe6b90ad4351df0b8443c3a3207d92c30f4ef70f3dd6"}
May 20 14:23:30.226709 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:30.226714    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm" event={"ID":"01a6cb98-37b5-49c2-a601-d09952a1df97","Type":"ContainerStarted","Data":"f566fc3a260675ef776d4c070acc0d20550710359f4c1c1bbe9081c45e79d9eb"}
May 20 14:23:30.227014 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:30.226886    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"
May 20 14:23:30.246951 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:30.246850    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm" podStartSLOduration=2.246834423 podStartE2EDuration="2.246834423s" podCreationTimestamp="2026-05-20 14:23:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:23:30.244795975 +0000 UTC m=+1977.482412657" watchObservedRunningTime="2026-05-20 14:23:30.246834423 +0000 UTC m=+1977.484451108"
May 20 14:23:31.229655 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:31.229605    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"
May 20 14:23:31.230957 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:31.230930    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm" podUID="01a6cb98-37b5-49c2-a601-d09952a1df97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
May 20 14:23:31.515052 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:31.514954    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt" podUID="881877ca-fd37-439c-b536-73be79de984c" containerName="sequence-graph-734fc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:23:31.874528 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:31.874498    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"
May 20 14:23:31.938508 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:31.938468    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mlp5\" (UniqueName: \"kubernetes.io/projected/7bb62959-33c7-4cf3-824a-616ee8f239e2-kube-api-access-2mlp5\") pod \"7bb62959-33c7-4cf3-824a-616ee8f239e2\" (UID: \"7bb62959-33c7-4cf3-824a-616ee8f239e2\") "
May 20 14:23:31.938704 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:31.938520    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-734fc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bb62959-33c7-4cf3-824a-616ee8f239e2-error-404-isvc-734fc-kube-rbac-proxy-sar-config\") pod \"7bb62959-33c7-4cf3-824a-616ee8f239e2\" (UID: \"7bb62959-33c7-4cf3-824a-616ee8f239e2\") "
May 20 14:23:31.938704 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:31.938595    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bb62959-33c7-4cf3-824a-616ee8f239e2-proxy-tls\") pod \"7bb62959-33c7-4cf3-824a-616ee8f239e2\" (UID: \"7bb62959-33c7-4cf3-824a-616ee8f239e2\") "
May 20 14:23:31.938931 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:31.938905    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb62959-33c7-4cf3-824a-616ee8f239e2-error-404-isvc-734fc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-734fc-kube-rbac-proxy-sar-config") pod "7bb62959-33c7-4cf3-824a-616ee8f239e2" (UID: "7bb62959-33c7-4cf3-824a-616ee8f239e2"). InnerVolumeSpecName "error-404-isvc-734fc-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:23:31.940768 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:31.940742    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb62959-33c7-4cf3-824a-616ee8f239e2-kube-api-access-2mlp5" (OuterVolumeSpecName: "kube-api-access-2mlp5") pod "7bb62959-33c7-4cf3-824a-616ee8f239e2" (UID: "7bb62959-33c7-4cf3-824a-616ee8f239e2"). InnerVolumeSpecName "kube-api-access-2mlp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:23:31.940854 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:31.940814    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bb62959-33c7-4cf3-824a-616ee8f239e2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7bb62959-33c7-4cf3-824a-616ee8f239e2" (UID: "7bb62959-33c7-4cf3-824a-616ee8f239e2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:23:32.039690 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:32.039557    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mlp5\" (UniqueName: \"kubernetes.io/projected/7bb62959-33c7-4cf3-824a-616ee8f239e2-kube-api-access-2mlp5\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:23:32.039690 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:32.039595    2570 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-734fc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bb62959-33c7-4cf3-824a-616ee8f239e2-error-404-isvc-734fc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:23:32.039690 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:32.039636    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bb62959-33c7-4cf3-824a-616ee8f239e2-proxy-tls\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:23:32.235029 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:32.234997    2570 generic.go:358] "Generic (PLEG): container finished" podID="7bb62959-33c7-4cf3-824a-616ee8f239e2" containerID="3b077e31529d7078dcc0f4d7557b123050863156e7d01674a9eab49c28a7ff1e" exitCode=0
May 20 14:23:32.235470 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:32.235070    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"
May 20 14:23:32.235470 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:32.235081    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w" event={"ID":"7bb62959-33c7-4cf3-824a-616ee8f239e2","Type":"ContainerDied","Data":"3b077e31529d7078dcc0f4d7557b123050863156e7d01674a9eab49c28a7ff1e"}
May 20 14:23:32.235470 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:32.235116    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w" event={"ID":"7bb62959-33c7-4cf3-824a-616ee8f239e2","Type":"ContainerDied","Data":"d5ff8aa537da36dc27cccd9a75291d97fe53fba0b2bb87bec6ace99b33f20caf"}
May 20 14:23:32.235470 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:32.235132    2570 scope.go:117] "RemoveContainer" containerID="1d4eb9d5e21f33468d91cbedb36c06b2fe1afd09615cbe34382c4790a83e9b34"
May 20 14:23:32.235673 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:32.235650    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm" podUID="01a6cb98-37b5-49c2-a601-d09952a1df97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
May 20 14:23:32.243844 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:32.243827    2570 scope.go:117] "RemoveContainer" containerID="3b077e31529d7078dcc0f4d7557b123050863156e7d01674a9eab49c28a7ff1e"
May 20 14:23:32.251781 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:32.251763    2570 scope.go:117] "RemoveContainer" containerID="1d4eb9d5e21f33468d91cbedb36c06b2fe1afd09615cbe34382c4790a83e9b34"
May 20 14:23:32.252063 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:23:32.252044    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d4eb9d5e21f33468d91cbedb36c06b2fe1afd09615cbe34382c4790a83e9b34\": container with ID starting with 1d4eb9d5e21f33468d91cbedb36c06b2fe1afd09615cbe34382c4790a83e9b34 not found: ID does not exist" containerID="1d4eb9d5e21f33468d91cbedb36c06b2fe1afd09615cbe34382c4790a83e9b34"
May 20 14:23:32.252122 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:32.252071    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d4eb9d5e21f33468d91cbedb36c06b2fe1afd09615cbe34382c4790a83e9b34"} err="failed to get container status \"1d4eb9d5e21f33468d91cbedb36c06b2fe1afd09615cbe34382c4790a83e9b34\": rpc error: code = NotFound desc = could not find container \"1d4eb9d5e21f33468d91cbedb36c06b2fe1afd09615cbe34382c4790a83e9b34\": container with ID starting with 1d4eb9d5e21f33468d91cbedb36c06b2fe1afd09615cbe34382c4790a83e9b34 not found: ID does not exist"
May 20 14:23:32.252122 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:32.252089    2570 scope.go:117] "RemoveContainer" containerID="3b077e31529d7078dcc0f4d7557b123050863156e7d01674a9eab49c28a7ff1e"
May 20 14:23:32.252339 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:23:32.252319    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b077e31529d7078dcc0f4d7557b123050863156e7d01674a9eab49c28a7ff1e\": container with ID starting with 3b077e31529d7078dcc0f4d7557b123050863156e7d01674a9eab49c28a7ff1e not found: ID does not exist" containerID="3b077e31529d7078dcc0f4d7557b123050863156e7d01674a9eab49c28a7ff1e"
May 20 14:23:32.252386 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:32.252346    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b077e31529d7078dcc0f4d7557b123050863156e7d01674a9eab49c28a7ff1e"} err="failed to get container status \"3b077e31529d7078dcc0f4d7557b123050863156e7d01674a9eab49c28a7ff1e\": rpc error: code = NotFound desc = could not find container \"3b077e31529d7078dcc0f4d7557b123050863156e7d01674a9eab49c28a7ff1e\": container with ID starting with 3b077e31529d7078dcc0f4d7557b123050863156e7d01674a9eab49c28a7ff1e not found: ID does not exist"
May 20 14:23:32.259131 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:32.259109    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"]
May 20 14:23:32.266523 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:32.266498    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-734fc-predictor-5995d7675c-86m5w"]
May 20 14:23:33.363894 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:33.363856    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bb62959-33c7-4cf3-824a-616ee8f239e2" path="/var/lib/kubelet/pods/7bb62959-33c7-4cf3-824a-616ee8f239e2/volumes"
May 20 14:23:36.515764 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:36.515721    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt" podUID="881877ca-fd37-439c-b536-73be79de984c" containerName="sequence-graph-734fc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:23:37.122448 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:37.122405    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq" podUID="b151f8aa-7bf0-4a45-8612-a746025bcae7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 20 14:23:37.240005 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:37.239971    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"
May 20 14:23:37.240590 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:37.240539    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm" podUID="01a6cb98-37b5-49c2-a601-d09952a1df97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
May 20 14:23:41.514869 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:41.514823    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt" podUID="881877ca-fd37-439c-b536-73be79de984c" containerName="sequence-graph-734fc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:23:41.515314 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:41.514962    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt"
May 20 14:23:46.515695 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:46.515643    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt" podUID="881877ca-fd37-439c-b536-73be79de984c" containerName="sequence-graph-734fc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:23:47.123353 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:47.123324    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"
May 20 14:23:47.240528 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:47.240489    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm" podUID="01a6cb98-37b5-49c2-a601-d09952a1df97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
May 20 14:23:51.515512 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:51.515471    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt" podUID="881877ca-fd37-439c-b536-73be79de984c" containerName="sequence-graph-734fc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:23:56.515139 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:56.515101    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt" podUID="881877ca-fd37-439c-b536-73be79de984c" containerName="sequence-graph-734fc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:23:57.240895 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:57.240848    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm" podUID="01a6cb98-37b5-49c2-a601-d09952a1df97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
May 20 14:23:58.565216 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.565191    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt"
May 20 14:23:58.683143 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.683041    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/881877ca-fd37-439c-b536-73be79de984c-proxy-tls\") pod \"881877ca-fd37-439c-b536-73be79de984c\" (UID: \"881877ca-fd37-439c-b536-73be79de984c\") "
May 20 14:23:58.683143 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.683084    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/881877ca-fd37-439c-b536-73be79de984c-openshift-service-ca-bundle\") pod \"881877ca-fd37-439c-b536-73be79de984c\" (UID: \"881877ca-fd37-439c-b536-73be79de984c\") "
May 20 14:23:58.683484 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.683458    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/881877ca-fd37-439c-b536-73be79de984c-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "881877ca-fd37-439c-b536-73be79de984c" (UID: "881877ca-fd37-439c-b536-73be79de984c"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:23:58.685341 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.685318    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/881877ca-fd37-439c-b536-73be79de984c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "881877ca-fd37-439c-b536-73be79de984c" (UID: "881877ca-fd37-439c-b536-73be79de984c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:23:58.784144 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.784106    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/881877ca-fd37-439c-b536-73be79de984c-proxy-tls\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:23:58.784144 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.784140    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/881877ca-fd37-439c-b536-73be79de984c-openshift-service-ca-bundle\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:23:58.863959 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.863915    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495"]
May 20 14:23:58.864376 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.864356    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb62959-33c7-4cf3-824a-616ee8f239e2" containerName="kserve-container"
May 20 14:23:58.864465 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.864378    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb62959-33c7-4cf3-824a-616ee8f239e2" containerName="kserve-container"
May 20 14:23:58.864465 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.864388    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb62959-33c7-4cf3-824a-616ee8f239e2" containerName="kube-rbac-proxy"
May 20 14:23:58.864465 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.864396    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb62959-33c7-4cf3-824a-616ee8f239e2" containerName="kube-rbac-proxy"
May 20 14:23:58.864465 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.864436    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="881877ca-fd37-439c-b536-73be79de984c" containerName="sequence-graph-734fc"
May 20 14:23:58.864465 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.864446    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="881877ca-fd37-439c-b536-73be79de984c" containerName="sequence-graph-734fc"
May 20 14:23:58.864802 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.864520    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="7bb62959-33c7-4cf3-824a-616ee8f239e2" containerName="kube-rbac-proxy"
May 20 14:23:58.864802 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.864536    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="7bb62959-33c7-4cf3-824a-616ee8f239e2" containerName="kserve-container"
May 20 14:23:58.864802 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.864554    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="881877ca-fd37-439c-b536-73be79de984c" containerName="sequence-graph-734fc"
May 20 14:23:58.867939 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.867917    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495"
May 20 14:23:58.870252 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.870224    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-a868a-serving-cert\""
May 20 14:23:58.870378 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.870233    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-a868a-kube-rbac-proxy-sar-config\""
May 20 14:23:58.879606 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.879559    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495"]
May 20 14:23:58.985563 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.985472    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea4f12b1-b727-4900-b2ea-5a7afc7bd513-openshift-service-ca-bundle\") pod \"splitter-graph-a868a-757c8c8c4d-92495\" (UID: \"ea4f12b1-b727-4900-b2ea-5a7afc7bd513\") " pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495"
May 20 14:23:58.985563 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:58.985542    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea4f12b1-b727-4900-b2ea-5a7afc7bd513-proxy-tls\") pod \"splitter-graph-a868a-757c8c8c4d-92495\" (UID: \"ea4f12b1-b727-4900-b2ea-5a7afc7bd513\") " pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495"
May 20 14:23:59.086903 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:59.086861    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea4f12b1-b727-4900-b2ea-5a7afc7bd513-proxy-tls\") pod \"splitter-graph-a868a-757c8c8c4d-92495\" (UID: \"ea4f12b1-b727-4900-b2ea-5a7afc7bd513\") " pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495"
May 20 14:23:59.087089 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:59.086962    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea4f12b1-b727-4900-b2ea-5a7afc7bd513-openshift-service-ca-bundle\") pod \"splitter-graph-a868a-757c8c8c4d-92495\" (UID: \"ea4f12b1-b727-4900-b2ea-5a7afc7bd513\") " pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495"
May 20 14:23:59.087717 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:59.087689    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea4f12b1-b727-4900-b2ea-5a7afc7bd513-openshift-service-ca-bundle\") pod \"splitter-graph-a868a-757c8c8c4d-92495\" (UID: \"ea4f12b1-b727-4900-b2ea-5a7afc7bd513\") " pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495"
May 20 14:23:59.089557 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:59.089539    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea4f12b1-b727-4900-b2ea-5a7afc7bd513-proxy-tls\") pod \"splitter-graph-a868a-757c8c8c4d-92495\" (UID: \"ea4f12b1-b727-4900-b2ea-5a7afc7bd513\") " pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495"
May 20 14:23:59.186697 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:59.186656    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495"
May 20 14:23:59.324852 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:59.324828    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495"]
May 20 14:23:59.327122 ip-10-0-136-75 kubenswrapper[2570]: W0520 14:23:59.327093    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4f12b1_b727_4900_b2ea_5a7afc7bd513.slice/crio-2b2aaea603cb1aba902e9651a41c5a7b6d2c09ddb1ac0c3a1c350073f26d5671 WatchSource:0}: Error finding container 2b2aaea603cb1aba902e9651a41c5a7b6d2c09ddb1ac0c3a1c350073f26d5671: Status 404 returned error can't find the container with id 2b2aaea603cb1aba902e9651a41c5a7b6d2c09ddb1ac0c3a1c350073f26d5671
May 20 14:23:59.340520 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:59.340491    2570 generic.go:358] "Generic (PLEG): container finished" podID="881877ca-fd37-439c-b536-73be79de984c" containerID="1c1f0131c1b7c9d67bdb14fde0ca971c1e77bb4cad94b69cdb159c5f9e76a6cc" exitCode=0
May 20 14:23:59.340664 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:59.340575    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt"
May 20 14:23:59.340664 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:59.340569    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt" event={"ID":"881877ca-fd37-439c-b536-73be79de984c","Type":"ContainerDied","Data":"1c1f0131c1b7c9d67bdb14fde0ca971c1e77bb4cad94b69cdb159c5f9e76a6cc"}
May 20 14:23:59.340774 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:59.340663    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt" event={"ID":"881877ca-fd37-439c-b536-73be79de984c","Type":"ContainerDied","Data":"c26d2ef28cdb5ebe2f116a48cb26139d2030e9a5fb8c49921290089a231ff246"}
May 20 14:23:59.340774 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:59.340686    2570 scope.go:117] "RemoveContainer" containerID="1c1f0131c1b7c9d67bdb14fde0ca971c1e77bb4cad94b69cdb159c5f9e76a6cc"
May 20 14:23:59.345537 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:59.345496    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495" event={"ID":"ea4f12b1-b727-4900-b2ea-5a7afc7bd513","Type":"ContainerStarted","Data":"2b2aaea603cb1aba902e9651a41c5a7b6d2c09ddb1ac0c3a1c350073f26d5671"}
May 20 14:23:59.355265 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:59.355241    2570 scope.go:117] "RemoveContainer" containerID="1c1f0131c1b7c9d67bdb14fde0ca971c1e77bb4cad94b69cdb159c5f9e76a6cc"
May 20 14:23:59.355599 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:23:59.355580    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c1f0131c1b7c9d67bdb14fde0ca971c1e77bb4cad94b69cdb159c5f9e76a6cc\": container with ID starting with 1c1f0131c1b7c9d67bdb14fde0ca971c1e77bb4cad94b69cdb159c5f9e76a6cc not found: ID does not exist" containerID="1c1f0131c1b7c9d67bdb14fde0ca971c1e77bb4cad94b69cdb159c5f9e76a6cc"
May 20 14:23:59.355709 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:59.355630    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c1f0131c1b7c9d67bdb14fde0ca971c1e77bb4cad94b69cdb159c5f9e76a6cc"} err="failed to get container status \"1c1f0131c1b7c9d67bdb14fde0ca971c1e77bb4cad94b69cdb159c5f9e76a6cc\": rpc error: code = NotFound desc = could not find container \"1c1f0131c1b7c9d67bdb14fde0ca971c1e77bb4cad94b69cdb159c5f9e76a6cc\": container with ID starting with 1c1f0131c1b7c9d67bdb14fde0ca971c1e77bb4cad94b69cdb159c5f9e76a6cc not found: ID does not exist"
May 20 14:23:59.376188 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:59.376163    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt"]
May 20 14:23:59.380587 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:23:59.380558    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-734fc-699db48bbd-8gvbt"]
May 20 14:24:00.349933 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:00.349891    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495" event={"ID":"ea4f12b1-b727-4900-b2ea-5a7afc7bd513","Type":"ContainerStarted","Data":"35127c9b2c33bd42276e0749cdbe5f283b3dab8051d7fe5796bf6d5a1ed2aa32"}
May 20 14:24:00.350422 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:00.349948    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495"
May 20 14:24:00.369529 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:00.369479    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495" podStartSLOduration=2.369465697 podStartE2EDuration="2.369465697s" podCreationTimestamp="2026-05-20 14:23:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:24:00.367845355 +0000 UTC m=+2007.605461997" watchObservedRunningTime="2026-05-20 14:24:00.369465697 +0000 UTC m=+2007.607082344"
May 20 14:24:01.363934 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:01.363900    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="881877ca-fd37-439c-b536-73be79de984c" path="/var/lib/kubelet/pods/881877ca-fd37-439c-b536-73be79de984c/volumes"
May 20 14:24:06.359709 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:06.359674    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495"
May 20 14:24:07.240659 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:07.240592    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm" podUID="01a6cb98-37b5-49c2-a601-d09952a1df97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
May 20 14:24:08.964057 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:08.963975    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495"]
May 20 14:24:08.964483 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:08.964192    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495" podUID="ea4f12b1-b727-4900-b2ea-5a7afc7bd513" containerName="splitter-graph-a868a" containerID="cri-o://35127c9b2c33bd42276e0749cdbe5f283b3dab8051d7fe5796bf6d5a1ed2aa32" gracePeriod=30
May 20 14:24:09.205886 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:09.202163    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"]
May 20 14:24:09.205886 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:09.202655    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq" podUID="b151f8aa-7bf0-4a45-8612-a746025bcae7" containerName="kserve-container" containerID="cri-o://805f9854eaa2ce27ee93211de65a00399bca7f465b26aad765be6dda1f399073" gracePeriod=30
May 20 14:24:09.205886 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:09.202846    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq" podUID="b151f8aa-7bf0-4a45-8612-a746025bcae7" containerName="kube-rbac-proxy" containerID="cri-o://b1ae5c6e014f8b2d00d6ad72d7001ec698fa0d49da408283e5316231b3d45dda" gracePeriod=30
May 20 14:24:09.262425 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:09.262391    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"]
May 20 14:24:09.266377 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:09.266354    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"
May 20 14:24:09.268908 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:09.268888    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-31b2e-predictor-serving-cert\""
May 20 14:24:09.269024 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:09.268913    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-31b2e-kube-rbac-proxy-sar-config\""
May 20 14:24:09.280160 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:09.280129    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"]
May 20 14:24:09.381867 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:09.381827    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-31b2e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/247807be-896a-45f3-9ae9-5266e0b9fa63-error-404-isvc-31b2e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-31b2e-predictor-848cf47897-zpj2x\" (UID: \"247807be-896a-45f3-9ae9-5266e0b9fa63\") " pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"
May 20 14:24:09.382038 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:09.381954    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp5nj\" (UniqueName: \"kubernetes.io/projected/247807be-896a-45f3-9ae9-5266e0b9fa63-kube-api-access-wp5nj\") pod \"error-404-isvc-31b2e-predictor-848cf47897-zpj2x\" (UID: \"247807be-896a-45f3-9ae9-5266e0b9fa63\") " pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"
May 20 14:24:09.382038 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:09.381985    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/247807be-896a-45f3-9ae9-5266e0b9fa63-proxy-tls\") pod \"error-404-isvc-31b2e-predictor-848cf47897-zpj2x\" (UID: \"247807be-896a-45f3-9ae9-5266e0b9fa63\") " pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"
May 20 14:24:09.392394 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:09.392366    2570 generic.go:358] "Generic (PLEG): container finished" podID="b151f8aa-7bf0-4a45-8612-a746025bcae7" containerID="b1ae5c6e014f8b2d00d6ad72d7001ec698fa0d49da408283e5316231b3d45dda" exitCode=2
May 20 14:24:09.392529 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:09.392400    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq" event={"ID":"b151f8aa-7bf0-4a45-8612-a746025bcae7","Type":"ContainerDied","Data":"b1ae5c6e014f8b2d00d6ad72d7001ec698fa0d49da408283e5316231b3d45dda"}
May 20 14:24:09.482730 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:09.482640    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-31b2e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/247807be-896a-45f3-9ae9-5266e0b9fa63-error-404-isvc-31b2e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-31b2e-predictor-848cf47897-zpj2x\" (UID: \"247807be-896a-45f3-9ae9-5266e0b9fa63\") " pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"
May 20 14:24:09.482905 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:09.482768    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wp5nj\" (UniqueName: \"kubernetes.io/projected/247807be-896a-45f3-9ae9-5266e0b9fa63-kube-api-access-wp5nj\") pod \"error-404-isvc-31b2e-predictor-848cf47897-zpj2x\" (UID: \"247807be-896a-45f3-9ae9-5266e0b9fa63\") " pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"
May 20 14:24:09.482905 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:09.482807    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/247807be-896a-45f3-9ae9-5266e0b9fa63-proxy-tls\") pod \"error-404-isvc-31b2e-predictor-848cf47897-zpj2x\" (UID: \"247807be-896a-45f3-9ae9-5266e0b9fa63\") " pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"
May 20 14:24:09.483291 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:09.483268    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-31b2e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/247807be-896a-45f3-9ae9-5266e0b9fa63-error-404-isvc-31b2e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-31b2e-predictor-848cf47897-zpj2x\" (UID: \"247807be-896a-45f3-9ae9-5266e0b9fa63\") " pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"
May 20 14:24:09.485409 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:09.485379    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/247807be-896a-45f3-9ae9-5266e0b9fa63-proxy-tls\") pod \"error-404-isvc-31b2e-predictor-848cf47897-zpj2x\" (UID: \"247807be-896a-45f3-9ae9-5266e0b9fa63\") " pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"
May 20 14:24:09.492330 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:09.492307    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp5nj\" (UniqueName: \"kubernetes.io/projected/247807be-896a-45f3-9ae9-5266e0b9fa63-kube-api-access-wp5nj\") pod \"error-404-isvc-31b2e-predictor-848cf47897-zpj2x\" (UID: \"247807be-896a-45f3-9ae9-5266e0b9fa63\") " pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"
May 20 14:24:09.579287 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:09.579251    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"
May 20 14:24:09.721435 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:09.721394    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"]
May 20 14:24:09.725414 ip-10-0-136-75 kubenswrapper[2570]: W0520 14:24:09.725388    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247807be_896a_45f3_9ae9_5266e0b9fa63.slice/crio-4ad103ab757ca92ea3e511f26f77768c42c4b5105c4ae4fc1611c9a54ab1727e WatchSource:0}: Error finding container 4ad103ab757ca92ea3e511f26f77768c42c4b5105c4ae4fc1611c9a54ab1727e: Status 404 returned error can't find the container with id 4ad103ab757ca92ea3e511f26f77768c42c4b5105c4ae4fc1611c9a54ab1727e
May 20 14:24:10.398055 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:10.398018    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x" event={"ID":"247807be-896a-45f3-9ae9-5266e0b9fa63","Type":"ContainerStarted","Data":"a006e73c235244dd4242d44b0a91dda7a7d28334c893b5d5ca7269e505bc48a4"}
May 20 14:24:10.398055 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:10.398057    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x" event={"ID":"247807be-896a-45f3-9ae9-5266e0b9fa63","Type":"ContainerStarted","Data":"146bc6df8bdd4526a74f4c415d445db6c7cb308b5632003b387d3954f6f33cba"}
May 20 14:24:10.398055 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:10.398067    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x" event={"ID":"247807be-896a-45f3-9ae9-5266e0b9fa63","Type":"ContainerStarted","Data":"4ad103ab757ca92ea3e511f26f77768c42c4b5105c4ae4fc1611c9a54ab1727e"}
May 20 14:24:10.398525 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:10.398104    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"
May 20 14:24:10.422467 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:10.422413    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x" podStartSLOduration=1.422396908 podStartE2EDuration="1.422396908s" podCreationTimestamp="2026-05-20 14:24:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:24:10.41935621 +0000 UTC m=+2017.656972857" watchObservedRunningTime="2026-05-20 14:24:10.422396908 +0000 UTC m=+2017.660013554"
May 20 14:24:11.359037 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:11.358998    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495" podUID="ea4f12b1-b727-4900-b2ea-5a7afc7bd513" containerName="splitter-graph-a868a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:24:11.401865 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:11.401512    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"
May 20 14:24:11.402818 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:11.402795    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x" podUID="247807be-896a-45f3-9ae9-5266e0b9fa63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
May 20 14:24:12.117955 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:12.117914    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq" podUID="b151f8aa-7bf0-4a45-8612-a746025bcae7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
May 20 14:24:12.405319 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:12.405269    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x" podUID="247807be-896a-45f3-9ae9-5266e0b9fa63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
May 20 14:24:12.576154 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:12.576124    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"
May 20 14:24:12.612931 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:12.612894    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b151f8aa-7bf0-4a45-8612-a746025bcae7-proxy-tls\") pod \"b151f8aa-7bf0-4a45-8612-a746025bcae7\" (UID: \"b151f8aa-7bf0-4a45-8612-a746025bcae7\") "
May 20 14:24:12.613127 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:12.612957    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj78s\" (UniqueName: \"kubernetes.io/projected/b151f8aa-7bf0-4a45-8612-a746025bcae7-kube-api-access-nj78s\") pod \"b151f8aa-7bf0-4a45-8612-a746025bcae7\" (UID: \"b151f8aa-7bf0-4a45-8612-a746025bcae7\") "
May 20 14:24:12.613127 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:12.612987    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-a868a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b151f8aa-7bf0-4a45-8612-a746025bcae7-error-404-isvc-a868a-kube-rbac-proxy-sar-config\") pod \"b151f8aa-7bf0-4a45-8612-a746025bcae7\" (UID: \"b151f8aa-7bf0-4a45-8612-a746025bcae7\") "
May 20 14:24:12.613458 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:12.613427    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b151f8aa-7bf0-4a45-8612-a746025bcae7-error-404-isvc-a868a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-a868a-kube-rbac-proxy-sar-config") pod "b151f8aa-7bf0-4a45-8612-a746025bcae7" (UID: "b151f8aa-7bf0-4a45-8612-a746025bcae7"). InnerVolumeSpecName "error-404-isvc-a868a-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:24:12.615134 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:12.615110    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b151f8aa-7bf0-4a45-8612-a746025bcae7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b151f8aa-7bf0-4a45-8612-a746025bcae7" (UID: "b151f8aa-7bf0-4a45-8612-a746025bcae7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:24:12.615229 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:12.615131    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b151f8aa-7bf0-4a45-8612-a746025bcae7-kube-api-access-nj78s" (OuterVolumeSpecName: "kube-api-access-nj78s") pod "b151f8aa-7bf0-4a45-8612-a746025bcae7" (UID: "b151f8aa-7bf0-4a45-8612-a746025bcae7"). InnerVolumeSpecName "kube-api-access-nj78s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:24:12.714062 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:12.713971    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj78s\" (UniqueName: \"kubernetes.io/projected/b151f8aa-7bf0-4a45-8612-a746025bcae7-kube-api-access-nj78s\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:24:12.714062 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:12.714007    2570 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-a868a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b151f8aa-7bf0-4a45-8612-a746025bcae7-error-404-isvc-a868a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:24:12.714062 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:12.714018    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b151f8aa-7bf0-4a45-8612-a746025bcae7-proxy-tls\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:24:13.409839 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:13.409796    2570 generic.go:358] "Generic (PLEG): container finished" podID="b151f8aa-7bf0-4a45-8612-a746025bcae7" containerID="805f9854eaa2ce27ee93211de65a00399bca7f465b26aad765be6dda1f399073" exitCode=0
May 20 14:24:13.410252 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:13.409876    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"
May 20 14:24:13.410252 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:13.409876    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq" event={"ID":"b151f8aa-7bf0-4a45-8612-a746025bcae7","Type":"ContainerDied","Data":"805f9854eaa2ce27ee93211de65a00399bca7f465b26aad765be6dda1f399073"}
May 20 14:24:13.410252 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:13.409976    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq" event={"ID":"b151f8aa-7bf0-4a45-8612-a746025bcae7","Type":"ContainerDied","Data":"35dc3ffc9650c41db51955d0e1623382c9a60fe5e3733db28dfdf50fd6ed0ec4"}
May 20 14:24:13.410252 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:13.409995    2570 scope.go:117] "RemoveContainer" containerID="b1ae5c6e014f8b2d00d6ad72d7001ec698fa0d49da408283e5316231b3d45dda"
May 20 14:24:13.418084 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:13.418067    2570 scope.go:117] "RemoveContainer" containerID="805f9854eaa2ce27ee93211de65a00399bca7f465b26aad765be6dda1f399073"
May 20 14:24:13.425376 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:13.425360    2570 scope.go:117] "RemoveContainer" containerID="b1ae5c6e014f8b2d00d6ad72d7001ec698fa0d49da408283e5316231b3d45dda"
May 20 14:24:13.425656 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:24:13.425635    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ae5c6e014f8b2d00d6ad72d7001ec698fa0d49da408283e5316231b3d45dda\": container with ID starting with b1ae5c6e014f8b2d00d6ad72d7001ec698fa0d49da408283e5316231b3d45dda not found: ID does not exist" containerID="b1ae5c6e014f8b2d00d6ad72d7001ec698fa0d49da408283e5316231b3d45dda"
May 20 14:24:13.425739 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:13.425668    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ae5c6e014f8b2d00d6ad72d7001ec698fa0d49da408283e5316231b3d45dda"} err="failed to get container status \"b1ae5c6e014f8b2d00d6ad72d7001ec698fa0d49da408283e5316231b3d45dda\": rpc error: code = NotFound desc = could not find container \"b1ae5c6e014f8b2d00d6ad72d7001ec698fa0d49da408283e5316231b3d45dda\": container with ID starting with b1ae5c6e014f8b2d00d6ad72d7001ec698fa0d49da408283e5316231b3d45dda not found: ID does not exist"
May 20 14:24:13.425739 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:13.425693    2570 scope.go:117] "RemoveContainer" containerID="805f9854eaa2ce27ee93211de65a00399bca7f465b26aad765be6dda1f399073"
May 20 14:24:13.425939 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:24:13.425924    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"805f9854eaa2ce27ee93211de65a00399bca7f465b26aad765be6dda1f399073\": container with ID starting with 805f9854eaa2ce27ee93211de65a00399bca7f465b26aad765be6dda1f399073 not found: ID does not exist" containerID="805f9854eaa2ce27ee93211de65a00399bca7f465b26aad765be6dda1f399073"
May 20 14:24:13.425995 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:13.425948    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"805f9854eaa2ce27ee93211de65a00399bca7f465b26aad765be6dda1f399073"} err="failed to get container status \"805f9854eaa2ce27ee93211de65a00399bca7f465b26aad765be6dda1f399073\": rpc error: code = NotFound desc = could not find container \"805f9854eaa2ce27ee93211de65a00399bca7f465b26aad765be6dda1f399073\": container with ID starting with 805f9854eaa2ce27ee93211de65a00399bca7f465b26aad765be6dda1f399073 not found: ID does not exist"
May 20 14:24:13.429454 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:13.429431    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"]
May 20 14:24:13.434469 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:13.434444    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a868a-predictor-848b5648c5-46cjq"]
May 20 14:24:15.363927 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:15.363889    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b151f8aa-7bf0-4a45-8612-a746025bcae7" path="/var/lib/kubelet/pods/b151f8aa-7bf0-4a45-8612-a746025bcae7/volumes"
May 20 14:24:16.358183 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:16.358143    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495" podUID="ea4f12b1-b727-4900-b2ea-5a7afc7bd513" containerName="splitter-graph-a868a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:24:17.241688 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:17.241657    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"
May 20 14:24:17.410292 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:17.410259    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"
May 20 14:24:17.410765 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:17.410737    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x" podUID="247807be-896a-45f3-9ae9-5266e0b9fa63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
May 20 14:24:21.358430 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:21.358390    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495" podUID="ea4f12b1-b727-4900-b2ea-5a7afc7bd513" containerName="splitter-graph-a868a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:24:21.358920 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:21.358503    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495"
May 20 14:24:26.358257 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:26.358215    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495" podUID="ea4f12b1-b727-4900-b2ea-5a7afc7bd513" containerName="splitter-graph-a868a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:24:27.411369 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:27.411323    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x" podUID="247807be-896a-45f3-9ae9-5266e0b9fa63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
May 20 14:24:28.660687 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:28.660650    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht"]
May 20 14:24:28.661138 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:28.661056    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b151f8aa-7bf0-4a45-8612-a746025bcae7" containerName="kserve-container"
May 20 14:24:28.661138 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:28.661069    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b151f8aa-7bf0-4a45-8612-a746025bcae7" containerName="kserve-container"
May 20 14:24:28.661138 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:28.661083    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b151f8aa-7bf0-4a45-8612-a746025bcae7" containerName="kube-rbac-proxy"
May 20 14:24:28.661138 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:28.661088    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b151f8aa-7bf0-4a45-8612-a746025bcae7" containerName="kube-rbac-proxy"
May 20 14:24:28.661276 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:28.661153    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="b151f8aa-7bf0-4a45-8612-a746025bcae7" containerName="kube-rbac-proxy"
May 20 14:24:28.661276 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:28.661162    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="b151f8aa-7bf0-4a45-8612-a746025bcae7" containerName="kserve-container"
May 20 14:24:28.665559 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:28.665538    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht"
May 20 14:24:28.669231 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:28.669198    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-3e303-serving-cert\""
May 20 14:24:28.669724 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:28.669705    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-3e303-kube-rbac-proxy-sar-config\""
May 20 14:24:28.677987 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:28.677954    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht"]
May 20 14:24:28.752461 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:28.752426    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14cab6c0-a554-416d-ae2e-76b2c089e696-proxy-tls\") pod \"switch-graph-3e303-66c47b978f-nfvht\" (UID: \"14cab6c0-a554-416d-ae2e-76b2c089e696\") " pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht"
May 20 14:24:28.752731 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:28.752511    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cab6c0-a554-416d-ae2e-76b2c089e696-openshift-service-ca-bundle\") pod \"switch-graph-3e303-66c47b978f-nfvht\" (UID: \"14cab6c0-a554-416d-ae2e-76b2c089e696\") " pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht"
May 20 14:24:28.853973 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:28.853936    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14cab6c0-a554-416d-ae2e-76b2c089e696-proxy-tls\") pod \"switch-graph-3e303-66c47b978f-nfvht\" (UID: \"14cab6c0-a554-416d-ae2e-76b2c089e696\") " pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht"
May 20 14:24:28.854146 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:28.853996    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cab6c0-a554-416d-ae2e-76b2c089e696-openshift-service-ca-bundle\") pod \"switch-graph-3e303-66c47b978f-nfvht\" (UID: \"14cab6c0-a554-416d-ae2e-76b2c089e696\") " pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht"
May 20 14:24:28.854146 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:24:28.854090    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-3e303-serving-cert: secret "switch-graph-3e303-serving-cert" not found
May 20 14:24:28.854232 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:24:28.854160    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14cab6c0-a554-416d-ae2e-76b2c089e696-proxy-tls podName:14cab6c0-a554-416d-ae2e-76b2c089e696 nodeName:}" failed. No retries permitted until 2026-05-20 14:24:29.354144114 +0000 UTC m=+2036.591760739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/14cab6c0-a554-416d-ae2e-76b2c089e696-proxy-tls") pod "switch-graph-3e303-66c47b978f-nfvht" (UID: "14cab6c0-a554-416d-ae2e-76b2c089e696") : secret "switch-graph-3e303-serving-cert" not found
May 20 14:24:28.854571 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:28.854555    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cab6c0-a554-416d-ae2e-76b2c089e696-openshift-service-ca-bundle\") pod \"switch-graph-3e303-66c47b978f-nfvht\" (UID: \"14cab6c0-a554-416d-ae2e-76b2c089e696\") " pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht"
May 20 14:24:29.358480 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:29.358428    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14cab6c0-a554-416d-ae2e-76b2c089e696-proxy-tls\") pod \"switch-graph-3e303-66c47b978f-nfvht\" (UID: \"14cab6c0-a554-416d-ae2e-76b2c089e696\") " pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht"
May 20 14:24:29.361107 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:29.361080    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14cab6c0-a554-416d-ae2e-76b2c089e696-proxy-tls\") pod \"switch-graph-3e303-66c47b978f-nfvht\" (UID: \"14cab6c0-a554-416d-ae2e-76b2c089e696\") " pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht"
May 20 14:24:29.576953 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:29.576913    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht"
May 20 14:24:29.716232 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:29.716205    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht"]
May 20 14:24:29.718595 ip-10-0-136-75 kubenswrapper[2570]: W0520 14:24:29.718563    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14cab6c0_a554_416d_ae2e_76b2c089e696.slice/crio-ee6d974d60e57b47133f99da4ebc0fabb931f440bb416ac3f891174ffc490bd8 WatchSource:0}: Error finding container ee6d974d60e57b47133f99da4ebc0fabb931f440bb416ac3f891174ffc490bd8: Status 404 returned error can't find the container with id ee6d974d60e57b47133f99da4ebc0fabb931f440bb416ac3f891174ffc490bd8
May 20 14:24:30.473243 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:30.473208    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht" event={"ID":"14cab6c0-a554-416d-ae2e-76b2c089e696","Type":"ContainerStarted","Data":"5b506e1811924852a113055bdba8d156d075081a9f90d6eaf7f47a925453ca63"}
May 20 14:24:30.473243 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:30.473242    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht" event={"ID":"14cab6c0-a554-416d-ae2e-76b2c089e696","Type":"ContainerStarted","Data":"ee6d974d60e57b47133f99da4ebc0fabb931f440bb416ac3f891174ffc490bd8"}
May 20 14:24:30.473482 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:30.473268    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht"
May 20 14:24:30.498663 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:30.498591    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht" podStartSLOduration=2.498576591 podStartE2EDuration="2.498576591s" podCreationTimestamp="2026-05-20 14:24:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:24:30.497082634 +0000 UTC m=+2037.734699292" watchObservedRunningTime="2026-05-20 14:24:30.498576591 +0000 UTC m=+2037.736193285"
May 20 14:24:31.358978 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:31.358927    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495" podUID="ea4f12b1-b727-4900-b2ea-5a7afc7bd513" containerName="splitter-graph-a868a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:24:36.358408 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:36.358367    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495" podUID="ea4f12b1-b727-4900-b2ea-5a7afc7bd513" containerName="splitter-graph-a868a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:24:36.486382 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:36.486349    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht"
May 20 14:24:37.411350 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:37.411311    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x" podUID="247807be-896a-45f3-9ae9-5266e0b9fa63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
May 20 14:24:39.013382 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:24:39.013346    2570 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4f12b1_b727_4900_b2ea_5a7afc7bd513.slice/crio-conmon-35127c9b2c33bd42276e0749cdbe5f283b3dab8051d7fe5796bf6d5a1ed2aa32.scope\": RecentStats: unable to find data in memory cache]"
May 20 14:24:39.119090 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:39.119061    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495"
May 20 14:24:39.247942 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:39.247841    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea4f12b1-b727-4900-b2ea-5a7afc7bd513-openshift-service-ca-bundle\") pod \"ea4f12b1-b727-4900-b2ea-5a7afc7bd513\" (UID: \"ea4f12b1-b727-4900-b2ea-5a7afc7bd513\") "
May 20 14:24:39.247942 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:39.247899    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea4f12b1-b727-4900-b2ea-5a7afc7bd513-proxy-tls\") pod \"ea4f12b1-b727-4900-b2ea-5a7afc7bd513\" (UID: \"ea4f12b1-b727-4900-b2ea-5a7afc7bd513\") "
May 20 14:24:39.248256 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:39.248231    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea4f12b1-b727-4900-b2ea-5a7afc7bd513-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ea4f12b1-b727-4900-b2ea-5a7afc7bd513" (UID: "ea4f12b1-b727-4900-b2ea-5a7afc7bd513"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:24:39.250257 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:39.250222    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea4f12b1-b727-4900-b2ea-5a7afc7bd513-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ea4f12b1-b727-4900-b2ea-5a7afc7bd513" (UID: "ea4f12b1-b727-4900-b2ea-5a7afc7bd513"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:24:39.348975 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:39.348940    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea4f12b1-b727-4900-b2ea-5a7afc7bd513-proxy-tls\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:24:39.348975 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:39.348972    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea4f12b1-b727-4900-b2ea-5a7afc7bd513-openshift-service-ca-bundle\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:24:39.506873 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:39.506791    2570 generic.go:358] "Generic (PLEG): container finished" podID="ea4f12b1-b727-4900-b2ea-5a7afc7bd513" containerID="35127c9b2c33bd42276e0749cdbe5f283b3dab8051d7fe5796bf6d5a1ed2aa32" exitCode=0
May 20 14:24:39.507035 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:39.506891    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495"
May 20 14:24:39.507035 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:39.506878    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495" event={"ID":"ea4f12b1-b727-4900-b2ea-5a7afc7bd513","Type":"ContainerDied","Data":"35127c9b2c33bd42276e0749cdbe5f283b3dab8051d7fe5796bf6d5a1ed2aa32"}
May 20 14:24:39.507035 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:39.507005    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495" event={"ID":"ea4f12b1-b727-4900-b2ea-5a7afc7bd513","Type":"ContainerDied","Data":"2b2aaea603cb1aba902e9651a41c5a7b6d2c09ddb1ac0c3a1c350073f26d5671"}
May 20 14:24:39.507035 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:39.507025    2570 scope.go:117] "RemoveContainer" containerID="35127c9b2c33bd42276e0749cdbe5f283b3dab8051d7fe5796bf6d5a1ed2aa32"
May 20 14:24:39.515500 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:39.515480    2570 scope.go:117] "RemoveContainer" containerID="35127c9b2c33bd42276e0749cdbe5f283b3dab8051d7fe5796bf6d5a1ed2aa32"
May 20 14:24:39.515791 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:24:39.515766    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35127c9b2c33bd42276e0749cdbe5f283b3dab8051d7fe5796bf6d5a1ed2aa32\": container with ID starting with 35127c9b2c33bd42276e0749cdbe5f283b3dab8051d7fe5796bf6d5a1ed2aa32 not found: ID does not exist" containerID="35127c9b2c33bd42276e0749cdbe5f283b3dab8051d7fe5796bf6d5a1ed2aa32"
May 20 14:24:39.515888 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:39.515798    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35127c9b2c33bd42276e0749cdbe5f283b3dab8051d7fe5796bf6d5a1ed2aa32"} err="failed to get container status \"35127c9b2c33bd42276e0749cdbe5f283b3dab8051d7fe5796bf6d5a1ed2aa32\": rpc error: code = NotFound desc = could not find container \"35127c9b2c33bd42276e0749cdbe5f283b3dab8051d7fe5796bf6d5a1ed2aa32\": container with ID starting with 35127c9b2c33bd42276e0749cdbe5f283b3dab8051d7fe5796bf6d5a1ed2aa32 not found: ID does not exist"
May 20 14:24:39.526052 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:39.526025    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495"]
May 20 14:24:39.530807 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:39.530783    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a868a-757c8c8c4d-92495"]
May 20 14:24:41.364330 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:41.364292    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea4f12b1-b727-4900-b2ea-5a7afc7bd513" path="/var/lib/kubelet/pods/ea4f12b1-b727-4900-b2ea-5a7afc7bd513/volumes"
May 20 14:24:47.411139 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:47.411098    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x" podUID="247807be-896a-45f3-9ae9-5266e0b9fa63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
May 20 14:24:57.412089 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:24:57.412013    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"
May 20 14:25:09.182534 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:09.182497    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff"]
May 20 14:25:09.182945 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:09.182928    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea4f12b1-b727-4900-b2ea-5a7afc7bd513" containerName="splitter-graph-a868a"
May 20 14:25:09.182945 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:09.182943    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea4f12b1-b727-4900-b2ea-5a7afc7bd513" containerName="splitter-graph-a868a"
May 20 14:25:09.183033 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:09.183018    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="ea4f12b1-b727-4900-b2ea-5a7afc7bd513" containerName="splitter-graph-a868a"
May 20 14:25:09.185910 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:09.185894    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff"
May 20 14:25:09.188432 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:09.188411    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-31b2e-kube-rbac-proxy-sar-config\""
May 20 14:25:09.188570 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:09.188430    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-31b2e-serving-cert\""
May 20 14:25:09.194634 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:09.194594    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff"]
May 20 14:25:09.209562 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:09.209532    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029acfcc-1456-4f49-85a1-a95d8ab677d1-openshift-service-ca-bundle\") pod \"splitter-graph-31b2e-5cc6cdf9c9-jndff\" (UID: \"029acfcc-1456-4f49-85a1-a95d8ab677d1\") " pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff"
May 20 14:25:09.209870 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:09.209846    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/029acfcc-1456-4f49-85a1-a95d8ab677d1-proxy-tls\") pod \"splitter-graph-31b2e-5cc6cdf9c9-jndff\" (UID: \"029acfcc-1456-4f49-85a1-a95d8ab677d1\") " pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff"
May 20 14:25:09.310807 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:09.310774    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029acfcc-1456-4f49-85a1-a95d8ab677d1-openshift-service-ca-bundle\") pod \"splitter-graph-31b2e-5cc6cdf9c9-jndff\" (UID: \"029acfcc-1456-4f49-85a1-a95d8ab677d1\") " pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff"
May 20 14:25:09.310990 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:09.310845    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/029acfcc-1456-4f49-85a1-a95d8ab677d1-proxy-tls\") pod \"splitter-graph-31b2e-5cc6cdf9c9-jndff\" (UID: \"029acfcc-1456-4f49-85a1-a95d8ab677d1\") " pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff"
May 20 14:25:09.310990 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:25:09.310955    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-31b2e-serving-cert: secret "splitter-graph-31b2e-serving-cert" not found
May 20 14:25:09.311112 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:25:09.311037    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029acfcc-1456-4f49-85a1-a95d8ab677d1-proxy-tls podName:029acfcc-1456-4f49-85a1-a95d8ab677d1 nodeName:}" failed. No retries permitted until 2026-05-20 14:25:09.811021062 +0000 UTC m=+2077.048637687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/029acfcc-1456-4f49-85a1-a95d8ab677d1-proxy-tls") pod "splitter-graph-31b2e-5cc6cdf9c9-jndff" (UID: "029acfcc-1456-4f49-85a1-a95d8ab677d1") : secret "splitter-graph-31b2e-serving-cert" not found
May 20 14:25:09.311447 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:09.311422    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029acfcc-1456-4f49-85a1-a95d8ab677d1-openshift-service-ca-bundle\") pod \"splitter-graph-31b2e-5cc6cdf9c9-jndff\" (UID: \"029acfcc-1456-4f49-85a1-a95d8ab677d1\") " pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff"
May 20 14:25:09.815558 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:09.815511    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/029acfcc-1456-4f49-85a1-a95d8ab677d1-proxy-tls\") pod \"splitter-graph-31b2e-5cc6cdf9c9-jndff\" (UID: \"029acfcc-1456-4f49-85a1-a95d8ab677d1\") " pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff"
May 20 14:25:09.818087 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:09.818058    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/029acfcc-1456-4f49-85a1-a95d8ab677d1-proxy-tls\") pod \"splitter-graph-31b2e-5cc6cdf9c9-jndff\" (UID: \"029acfcc-1456-4f49-85a1-a95d8ab677d1\") " pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff"
May 20 14:25:10.097016 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:10.096930    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff"
May 20 14:25:10.235294 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:10.235269    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff"]
May 20 14:25:10.237874 ip-10-0-136-75 kubenswrapper[2570]: W0520 14:25:10.237841    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029acfcc_1456_4f49_85a1_a95d8ab677d1.slice/crio-123b9d8cb1cfcaa5349b42a666dc491cf9c773eacb18d96a2e1ea78a417486d4 WatchSource:0}: Error finding container 123b9d8cb1cfcaa5349b42a666dc491cf9c773eacb18d96a2e1ea78a417486d4: Status 404 returned error can't find the container with id 123b9d8cb1cfcaa5349b42a666dc491cf9c773eacb18d96a2e1ea78a417486d4
May 20 14:25:10.616986 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:10.616943    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff" event={"ID":"029acfcc-1456-4f49-85a1-a95d8ab677d1","Type":"ContainerStarted","Data":"2f30028800bd14cf318ac02b18b8138c169364a5fbbdd13817b847ade7c2610d"}
May 20 14:25:10.617196 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:10.616993    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff" event={"ID":"029acfcc-1456-4f49-85a1-a95d8ab677d1","Type":"ContainerStarted","Data":"123b9d8cb1cfcaa5349b42a666dc491cf9c773eacb18d96a2e1ea78a417486d4"}
May 20 14:25:10.617196 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:10.617057    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff"
May 20 14:25:10.636289 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:10.636225    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff" podStartSLOduration=1.636204727 podStartE2EDuration="1.636204727s" podCreationTimestamp="2026-05-20 14:25:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:25:10.634904896 +0000 UTC m=+2077.872521543" watchObservedRunningTime="2026-05-20 14:25:10.636204727 +0000 UTC m=+2077.873821375"
May 20 14:25:16.627886 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:16.627858    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff"
May 20 14:25:33.405749 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:33.405715    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 14:25:33.410059 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:25:33.410036    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 14:30:33.430439 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:30:33.430405    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 14:30:33.435800 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:30:33.435776    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 14:33:23.820677 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:23.820646    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff"]
May 20 14:33:23.823335 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:23.820940    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff" podUID="029acfcc-1456-4f49-85a1-a95d8ab677d1" containerName="splitter-graph-31b2e" containerID="cri-o://2f30028800bd14cf318ac02b18b8138c169364a5fbbdd13817b847ade7c2610d" gracePeriod=30
May 20 14:33:24.000964 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:24.000926    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"]
May 20 14:33:24.001271 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:24.001241    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x" podUID="247807be-896a-45f3-9ae9-5266e0b9fa63" containerName="kube-rbac-proxy" containerID="cri-o://a006e73c235244dd4242d44b0a91dda7a7d28334c893b5d5ca7269e505bc48a4" gracePeriod=30
May 20 14:33:24.001353 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:24.001224    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x" podUID="247807be-896a-45f3-9ae9-5266e0b9fa63" containerName="kserve-container" containerID="cri-o://146bc6df8bdd4526a74f4c415d445db6c7cb308b5632003b387d3954f6f33cba" gracePeriod=30
May 20 14:33:24.340048 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:24.340013    2570 generic.go:358] "Generic (PLEG): container finished" podID="247807be-896a-45f3-9ae9-5266e0b9fa63" containerID="a006e73c235244dd4242d44b0a91dda7a7d28334c893b5d5ca7269e505bc48a4" exitCode=2
May 20 14:33:24.340306 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:24.340089    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x" event={"ID":"247807be-896a-45f3-9ae9-5266e0b9fa63","Type":"ContainerDied","Data":"a006e73c235244dd4242d44b0a91dda7a7d28334c893b5d5ca7269e505bc48a4"}
May 20 14:33:26.625351 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:26.625307    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff" podUID="029acfcc-1456-4f49-85a1-a95d8ab677d1" containerName="splitter-graph-31b2e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:33:27.352093 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:27.352059    2570 generic.go:358] "Generic (PLEG): container finished" podID="247807be-896a-45f3-9ae9-5266e0b9fa63" containerID="146bc6df8bdd4526a74f4c415d445db6c7cb308b5632003b387d3954f6f33cba" exitCode=0
May 20 14:33:27.352255 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:27.352134    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x" event={"ID":"247807be-896a-45f3-9ae9-5266e0b9fa63","Type":"ContainerDied","Data":"146bc6df8bdd4526a74f4c415d445db6c7cb308b5632003b387d3954f6f33cba"}
May 20 14:33:27.367403 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:27.367375    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"
May 20 14:33:27.432101 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:27.432000    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-31b2e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/247807be-896a-45f3-9ae9-5266e0b9fa63-error-404-isvc-31b2e-kube-rbac-proxy-sar-config\") pod \"247807be-896a-45f3-9ae9-5266e0b9fa63\" (UID: \"247807be-896a-45f3-9ae9-5266e0b9fa63\") "
May 20 14:33:27.432101 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:27.432057    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/247807be-896a-45f3-9ae9-5266e0b9fa63-proxy-tls\") pod \"247807be-896a-45f3-9ae9-5266e0b9fa63\" (UID: \"247807be-896a-45f3-9ae9-5266e0b9fa63\") "
May 20 14:33:27.432295 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:27.432128    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wp5nj\" (UniqueName: \"kubernetes.io/projected/247807be-896a-45f3-9ae9-5266e0b9fa63-kube-api-access-wp5nj\") pod \"247807be-896a-45f3-9ae9-5266e0b9fa63\" (UID: \"247807be-896a-45f3-9ae9-5266e0b9fa63\") "
May 20 14:33:27.432429 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:27.432406    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247807be-896a-45f3-9ae9-5266e0b9fa63-error-404-isvc-31b2e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-31b2e-kube-rbac-proxy-sar-config") pod "247807be-896a-45f3-9ae9-5266e0b9fa63" (UID: "247807be-896a-45f3-9ae9-5266e0b9fa63"). InnerVolumeSpecName "error-404-isvc-31b2e-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:33:27.434515 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:27.434477    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247807be-896a-45f3-9ae9-5266e0b9fa63-kube-api-access-wp5nj" (OuterVolumeSpecName: "kube-api-access-wp5nj") pod "247807be-896a-45f3-9ae9-5266e0b9fa63" (UID: "247807be-896a-45f3-9ae9-5266e0b9fa63"). InnerVolumeSpecName "kube-api-access-wp5nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:33:27.434687 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:27.434554    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/247807be-896a-45f3-9ae9-5266e0b9fa63-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "247807be-896a-45f3-9ae9-5266e0b9fa63" (UID: "247807be-896a-45f3-9ae9-5266e0b9fa63"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:33:27.533458 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:27.533424    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp5nj\" (UniqueName: \"kubernetes.io/projected/247807be-896a-45f3-9ae9-5266e0b9fa63-kube-api-access-wp5nj\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:33:27.533458 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:27.533455    2570 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-31b2e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/247807be-896a-45f3-9ae9-5266e0b9fa63-error-404-isvc-31b2e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:33:27.533458 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:27.533466    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/247807be-896a-45f3-9ae9-5266e0b9fa63-proxy-tls\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:33:28.357120 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:28.357088    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"
May 20 14:33:28.357557 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:28.357087    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x" event={"ID":"247807be-896a-45f3-9ae9-5266e0b9fa63","Type":"ContainerDied","Data":"4ad103ab757ca92ea3e511f26f77768c42c4b5105c4ae4fc1611c9a54ab1727e"}
May 20 14:33:28.357557 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:28.357212    2570 scope.go:117] "RemoveContainer" containerID="a006e73c235244dd4242d44b0a91dda7a7d28334c893b5d5ca7269e505bc48a4"
May 20 14:33:28.366359 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:28.366337    2570 scope.go:117] "RemoveContainer" containerID="146bc6df8bdd4526a74f4c415d445db6c7cb308b5632003b387d3954f6f33cba"
May 20 14:33:28.382807 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:28.382775    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"]
May 20 14:33:28.387724 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:28.387684    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31b2e-predictor-848cf47897-zpj2x"]
May 20 14:33:29.363291 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:29.363259    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="247807be-896a-45f3-9ae9-5266e0b9fa63" path="/var/lib/kubelet/pods/247807be-896a-45f3-9ae9-5266e0b9fa63/volumes"
May 20 14:33:31.624813 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:31.624770    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff" podUID="029acfcc-1456-4f49-85a1-a95d8ab677d1" containerName="splitter-graph-31b2e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:33:36.625177 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:36.625133    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff" podUID="029acfcc-1456-4f49-85a1-a95d8ab677d1" containerName="splitter-graph-31b2e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:33:36.625680 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:36.625256    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff"
May 20 14:33:41.625185 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:41.625140    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff" podUID="029acfcc-1456-4f49-85a1-a95d8ab677d1" containerName="splitter-graph-31b2e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:33:46.624967 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:46.624920    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff" podUID="029acfcc-1456-4f49-85a1-a95d8ab677d1" containerName="splitter-graph-31b2e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:33:51.625319 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:51.625226    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff" podUID="029acfcc-1456-4f49-85a1-a95d8ab677d1" containerName="splitter-graph-31b2e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:33:53.984917 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:53.984892    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff"
May 20 14:33:54.074918 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:54.074818    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/029acfcc-1456-4f49-85a1-a95d8ab677d1-proxy-tls\") pod \"029acfcc-1456-4f49-85a1-a95d8ab677d1\" (UID: \"029acfcc-1456-4f49-85a1-a95d8ab677d1\") "
May 20 14:33:54.074918 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:54.074891    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029acfcc-1456-4f49-85a1-a95d8ab677d1-openshift-service-ca-bundle\") pod \"029acfcc-1456-4f49-85a1-a95d8ab677d1\" (UID: \"029acfcc-1456-4f49-85a1-a95d8ab677d1\") "
May 20 14:33:54.075304 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:54.075278    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/029acfcc-1456-4f49-85a1-a95d8ab677d1-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "029acfcc-1456-4f49-85a1-a95d8ab677d1" (UID: "029acfcc-1456-4f49-85a1-a95d8ab677d1"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:33:54.077217 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:54.077188    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/029acfcc-1456-4f49-85a1-a95d8ab677d1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "029acfcc-1456-4f49-85a1-a95d8ab677d1" (UID: "029acfcc-1456-4f49-85a1-a95d8ab677d1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:33:54.176035 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:54.175992    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/029acfcc-1456-4f49-85a1-a95d8ab677d1-proxy-tls\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:33:54.176035 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:54.176033    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029acfcc-1456-4f49-85a1-a95d8ab677d1-openshift-service-ca-bundle\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:33:54.452301 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:54.452203    2570 generic.go:358] "Generic (PLEG): container finished" podID="029acfcc-1456-4f49-85a1-a95d8ab677d1" containerID="2f30028800bd14cf318ac02b18b8138c169364a5fbbdd13817b847ade7c2610d" exitCode=0
May 20 14:33:54.452301 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:54.452271    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff"
May 20 14:33:54.452301 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:54.452287    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff" event={"ID":"029acfcc-1456-4f49-85a1-a95d8ab677d1","Type":"ContainerDied","Data":"2f30028800bd14cf318ac02b18b8138c169364a5fbbdd13817b847ade7c2610d"}
May 20 14:33:54.452564 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:54.452324    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff" event={"ID":"029acfcc-1456-4f49-85a1-a95d8ab677d1","Type":"ContainerDied","Data":"123b9d8cb1cfcaa5349b42a666dc491cf9c773eacb18d96a2e1ea78a417486d4"}
May 20 14:33:54.452564 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:54.452343    2570 scope.go:117] "RemoveContainer" containerID="2f30028800bd14cf318ac02b18b8138c169364a5fbbdd13817b847ade7c2610d"
May 20 14:33:54.461667 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:54.461649    2570 scope.go:117] "RemoveContainer" containerID="2f30028800bd14cf318ac02b18b8138c169364a5fbbdd13817b847ade7c2610d"
May 20 14:33:54.461995 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:33:54.461972    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f30028800bd14cf318ac02b18b8138c169364a5fbbdd13817b847ade7c2610d\": container with ID starting with 2f30028800bd14cf318ac02b18b8138c169364a5fbbdd13817b847ade7c2610d not found: ID does not exist" containerID="2f30028800bd14cf318ac02b18b8138c169364a5fbbdd13817b847ade7c2610d"
May 20 14:33:54.462050 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:54.462005    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f30028800bd14cf318ac02b18b8138c169364a5fbbdd13817b847ade7c2610d"} err="failed to get container status \"2f30028800bd14cf318ac02b18b8138c169364a5fbbdd13817b847ade7c2610d\": rpc error: code = NotFound desc = could not find container \"2f30028800bd14cf318ac02b18b8138c169364a5fbbdd13817b847ade7c2610d\": container with ID starting with 2f30028800bd14cf318ac02b18b8138c169364a5fbbdd13817b847ade7c2610d not found: ID does not exist"
May 20 14:33:54.475682 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:54.475638    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff"]
May 20 14:33:54.480253 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:54.480214    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-31b2e-5cc6cdf9c9-jndff"]
May 20 14:33:55.363971 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:33:55.363926    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="029acfcc-1456-4f49-85a1-a95d8ab677d1" path="/var/lib/kubelet/pods/029acfcc-1456-4f49-85a1-a95d8ab677d1/volumes"
May 20 14:35:33.453848 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:35:33.453750    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 14:35:33.460179 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:35:33.460156    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 14:40:33.480391 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:33.480359    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 14:40:33.486728 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:33.486706    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 14:40:47.953028 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:47.952990    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht"]
May 20 14:40:47.953557 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:47.953200    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht" podUID="14cab6c0-a554-416d-ae2e-76b2c089e696" containerName="switch-graph-3e303" containerID="cri-o://5b506e1811924852a113055bdba8d156d075081a9f90d6eaf7f47a925453ca63" gracePeriod=30
May 20 14:40:48.148171 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:48.148133    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"]
May 20 14:40:48.148546 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:48.148517    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm" podUID="01a6cb98-37b5-49c2-a601-d09952a1df97" containerName="kserve-container" containerID="cri-o://afcf040c1fc926c2cde9fe6b90ad4351df0b8443c3a3207d92c30f4ef70f3dd6" gracePeriod=30
May 20 14:40:48.148645 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:48.148539    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm" podUID="01a6cb98-37b5-49c2-a601-d09952a1df97" containerName="kube-rbac-proxy" containerID="cri-o://3380eb6d23c5226011e16a6450c34e764830590e714a6e091a4f434908d5ccd5" gracePeriod=30
May 20 14:40:48.853764 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:48.853719    2570 generic.go:358] "Generic (PLEG): container finished" podID="01a6cb98-37b5-49c2-a601-d09952a1df97" containerID="3380eb6d23c5226011e16a6450c34e764830590e714a6e091a4f434908d5ccd5" exitCode=2
May 20 14:40:48.853950 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:48.853787    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm" event={"ID":"01a6cb98-37b5-49c2-a601-d09952a1df97","Type":"ContainerDied","Data":"3380eb6d23c5226011e16a6450c34e764830590e714a6e091a4f434908d5ccd5"}
May 20 14:40:51.484690 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.484656    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht" podUID="14cab6c0-a554-416d-ae2e-76b2c089e696" containerName="switch-graph-3e303" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:40:51.499365 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.499342    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"
May 20 14:40:51.614791 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.614712    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngb6q\" (UniqueName: \"kubernetes.io/projected/01a6cb98-37b5-49c2-a601-d09952a1df97-kube-api-access-ngb6q\") pod \"01a6cb98-37b5-49c2-a601-d09952a1df97\" (UID: \"01a6cb98-37b5-49c2-a601-d09952a1df97\") "
May 20 14:40:51.614791 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.614782    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01a6cb98-37b5-49c2-a601-d09952a1df97-proxy-tls\") pod \"01a6cb98-37b5-49c2-a601-d09952a1df97\" (UID: \"01a6cb98-37b5-49c2-a601-d09952a1df97\") "
May 20 14:40:51.615026 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.614876    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-3e303-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01a6cb98-37b5-49c2-a601-d09952a1df97-error-404-isvc-3e303-kube-rbac-proxy-sar-config\") pod \"01a6cb98-37b5-49c2-a601-d09952a1df97\" (UID: \"01a6cb98-37b5-49c2-a601-d09952a1df97\") "
May 20 14:40:51.615247 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.615221    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a6cb98-37b5-49c2-a601-d09952a1df97-error-404-isvc-3e303-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-3e303-kube-rbac-proxy-sar-config") pod "01a6cb98-37b5-49c2-a601-d09952a1df97" (UID: "01a6cb98-37b5-49c2-a601-d09952a1df97"). InnerVolumeSpecName "error-404-isvc-3e303-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:40:51.617011 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.616987    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a6cb98-37b5-49c2-a601-d09952a1df97-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "01a6cb98-37b5-49c2-a601-d09952a1df97" (UID: "01a6cb98-37b5-49c2-a601-d09952a1df97"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:40:51.617011 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.616990    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a6cb98-37b5-49c2-a601-d09952a1df97-kube-api-access-ngb6q" (OuterVolumeSpecName: "kube-api-access-ngb6q") pod "01a6cb98-37b5-49c2-a601-d09952a1df97" (UID: "01a6cb98-37b5-49c2-a601-d09952a1df97"). InnerVolumeSpecName "kube-api-access-ngb6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:40:51.716342 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.716303    2570 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-3e303-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01a6cb98-37b5-49c2-a601-d09952a1df97-error-404-isvc-3e303-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:40:51.716342 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.716334    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngb6q\" (UniqueName: \"kubernetes.io/projected/01a6cb98-37b5-49c2-a601-d09952a1df97-kube-api-access-ngb6q\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:40:51.716342 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.716347    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01a6cb98-37b5-49c2-a601-d09952a1df97-proxy-tls\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:40:51.866791 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.866698    2570 generic.go:358] "Generic (PLEG): container finished" podID="01a6cb98-37b5-49c2-a601-d09952a1df97" containerID="afcf040c1fc926c2cde9fe6b90ad4351df0b8443c3a3207d92c30f4ef70f3dd6" exitCode=0
May 20 14:40:51.866791 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.866772    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"
May 20 14:40:51.867016 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.866788    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm" event={"ID":"01a6cb98-37b5-49c2-a601-d09952a1df97","Type":"ContainerDied","Data":"afcf040c1fc926c2cde9fe6b90ad4351df0b8443c3a3207d92c30f4ef70f3dd6"}
May 20 14:40:51.867016 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.866824    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm" event={"ID":"01a6cb98-37b5-49c2-a601-d09952a1df97","Type":"ContainerDied","Data":"f566fc3a260675ef776d4c070acc0d20550710359f4c1c1bbe9081c45e79d9eb"}
May 20 14:40:51.867016 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.866838    2570 scope.go:117] "RemoveContainer" containerID="3380eb6d23c5226011e16a6450c34e764830590e714a6e091a4f434908d5ccd5"
May 20 14:40:51.875796 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.875778    2570 scope.go:117] "RemoveContainer" containerID="afcf040c1fc926c2cde9fe6b90ad4351df0b8443c3a3207d92c30f4ef70f3dd6"
May 20 14:40:51.883324 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.883306    2570 scope.go:117] "RemoveContainer" containerID="3380eb6d23c5226011e16a6450c34e764830590e714a6e091a4f434908d5ccd5"
May 20 14:40:51.883570 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:40:51.883548    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3380eb6d23c5226011e16a6450c34e764830590e714a6e091a4f434908d5ccd5\": container with ID starting with 3380eb6d23c5226011e16a6450c34e764830590e714a6e091a4f434908d5ccd5 not found: ID does not exist" containerID="3380eb6d23c5226011e16a6450c34e764830590e714a6e091a4f434908d5ccd5"
May 20 14:40:51.883677 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.883583    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3380eb6d23c5226011e16a6450c34e764830590e714a6e091a4f434908d5ccd5"} err="failed to get container status \"3380eb6d23c5226011e16a6450c34e764830590e714a6e091a4f434908d5ccd5\": rpc error: code = NotFound desc = could not find container \"3380eb6d23c5226011e16a6450c34e764830590e714a6e091a4f434908d5ccd5\": container with ID starting with 3380eb6d23c5226011e16a6450c34e764830590e714a6e091a4f434908d5ccd5 not found: ID does not exist"
May 20 14:40:51.883677 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.883632    2570 scope.go:117] "RemoveContainer" containerID="afcf040c1fc926c2cde9fe6b90ad4351df0b8443c3a3207d92c30f4ef70f3dd6"
May 20 14:40:51.883866 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:40:51.883851    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afcf040c1fc926c2cde9fe6b90ad4351df0b8443c3a3207d92c30f4ef70f3dd6\": container with ID starting with afcf040c1fc926c2cde9fe6b90ad4351df0b8443c3a3207d92c30f4ef70f3dd6 not found: ID does not exist" containerID="afcf040c1fc926c2cde9fe6b90ad4351df0b8443c3a3207d92c30f4ef70f3dd6"
May 20 14:40:51.883906 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.883872    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afcf040c1fc926c2cde9fe6b90ad4351df0b8443c3a3207d92c30f4ef70f3dd6"} err="failed to get container status \"afcf040c1fc926c2cde9fe6b90ad4351df0b8443c3a3207d92c30f4ef70f3dd6\": rpc error: code = NotFound desc = could not find container \"afcf040c1fc926c2cde9fe6b90ad4351df0b8443c3a3207d92c30f4ef70f3dd6\": container with ID starting with afcf040c1fc926c2cde9fe6b90ad4351df0b8443c3a3207d92c30f4ef70f3dd6 not found: ID does not exist"
May 20 14:40:51.891050 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.891027    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"]
May 20 14:40:51.896008 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:51.895990    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3e303-predictor-7ccc759659-gfmqm"]
May 20 14:40:53.364018 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:53.363982    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01a6cb98-37b5-49c2-a601-d09952a1df97" path="/var/lib/kubelet/pods/01a6cb98-37b5-49c2-a601-d09952a1df97/volumes"
May 20 14:40:56.483826 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:40:56.483789    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht" podUID="14cab6c0-a554-416d-ae2e-76b2c089e696" containerName="switch-graph-3e303" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:41:01.484754 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:01.484707    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht" podUID="14cab6c0-a554-416d-ae2e-76b2c089e696" containerName="switch-graph-3e303" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:41:01.485155 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:01.484828    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht"
May 20 14:41:04.588762 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:04.588735    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3e303-66c47b978f-nfvht_14cab6c0-a554-416d-ae2e-76b2c089e696/switch-graph-3e303/0.log"
May 20 14:41:05.425407 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:05.425362    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3e303-66c47b978f-nfvht_14cab6c0-a554-416d-ae2e-76b2c089e696/switch-graph-3e303/0.log"
May 20 14:41:06.246328 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:06.246294    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3e303-66c47b978f-nfvht_14cab6c0-a554-416d-ae2e-76b2c089e696/switch-graph-3e303/0.log"
May 20 14:41:06.484229 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:06.484191    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht" podUID="14cab6c0-a554-416d-ae2e-76b2c089e696" containerName="switch-graph-3e303" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:41:07.082728 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:07.082688    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3e303-66c47b978f-nfvht_14cab6c0-a554-416d-ae2e-76b2c089e696/switch-graph-3e303/0.log"
May 20 14:41:07.894187 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:07.894157    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3e303-66c47b978f-nfvht_14cab6c0-a554-416d-ae2e-76b2c089e696/switch-graph-3e303/0.log"
May 20 14:41:08.711887 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:08.711855    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3e303-66c47b978f-nfvht_14cab6c0-a554-416d-ae2e-76b2c089e696/switch-graph-3e303/0.log"
May 20 14:41:09.518907 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:09.518865    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3e303-66c47b978f-nfvht_14cab6c0-a554-416d-ae2e-76b2c089e696/switch-graph-3e303/0.log"
May 20 14:41:10.345514 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:10.345488    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3e303-66c47b978f-nfvht_14cab6c0-a554-416d-ae2e-76b2c089e696/switch-graph-3e303/0.log"
May 20 14:41:11.156516 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:11.156475    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3e303-66c47b978f-nfvht_14cab6c0-a554-416d-ae2e-76b2c089e696/switch-graph-3e303/0.log"
May 20 14:41:11.484044 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:11.483944    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht" podUID="14cab6c0-a554-416d-ae2e-76b2c089e696" containerName="switch-graph-3e303" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:41:11.987073 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:11.987044    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3e303-66c47b978f-nfvht_14cab6c0-a554-416d-ae2e-76b2c089e696/switch-graph-3e303/0.log"
May 20 14:41:12.801997 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:12.801964    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3e303-66c47b978f-nfvht_14cab6c0-a554-416d-ae2e-76b2c089e696/switch-graph-3e303/0.log"
May 20 14:41:13.629760 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:13.629733    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-3e303-66c47b978f-nfvht_14cab6c0-a554-416d-ae2e-76b2c089e696/switch-graph-3e303/0.log"
May 20 14:41:16.483798 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:16.483760    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht" podUID="14cab6c0-a554-416d-ae2e-76b2c089e696" containerName="switch-graph-3e303" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:41:18.102857 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:18.102825    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht"
May 20 14:41:18.133391 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:18.133356    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14cab6c0-a554-416d-ae2e-76b2c089e696-proxy-tls\") pod \"14cab6c0-a554-416d-ae2e-76b2c089e696\" (UID: \"14cab6c0-a554-416d-ae2e-76b2c089e696\") "
May 20 14:41:18.134084 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:18.133480    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cab6c0-a554-416d-ae2e-76b2c089e696-openshift-service-ca-bundle\") pod \"14cab6c0-a554-416d-ae2e-76b2c089e696\" (UID: \"14cab6c0-a554-416d-ae2e-76b2c089e696\") "
May 20 14:41:18.134201 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:18.134096    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14cab6c0-a554-416d-ae2e-76b2c089e696-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "14cab6c0-a554-416d-ae2e-76b2c089e696" (UID: "14cab6c0-a554-416d-ae2e-76b2c089e696"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:41:18.136468 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:18.136438    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14cab6c0-a554-416d-ae2e-76b2c089e696-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "14cab6c0-a554-416d-ae2e-76b2c089e696" (UID: "14cab6c0-a554-416d-ae2e-76b2c089e696"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:41:18.234570 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:18.234484    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cab6c0-a554-416d-ae2e-76b2c089e696-openshift-service-ca-bundle\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:41:18.234570 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:18.234513    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14cab6c0-a554-416d-ae2e-76b2c089e696-proxy-tls\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\""
May 20 14:41:18.957828 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:18.957784    2570 generic.go:358] "Generic (PLEG): container finished" podID="14cab6c0-a554-416d-ae2e-76b2c089e696" containerID="5b506e1811924852a113055bdba8d156d075081a9f90d6eaf7f47a925453ca63" exitCode=0
May 20 14:41:18.958284 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:18.957845    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht"
May 20 14:41:18.958284 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:18.957878    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht" event={"ID":"14cab6c0-a554-416d-ae2e-76b2c089e696","Type":"ContainerDied","Data":"5b506e1811924852a113055bdba8d156d075081a9f90d6eaf7f47a925453ca63"}
May 20 14:41:18.958284 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:18.957934    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht" event={"ID":"14cab6c0-a554-416d-ae2e-76b2c089e696","Type":"ContainerDied","Data":"ee6d974d60e57b47133f99da4ebc0fabb931f440bb416ac3f891174ffc490bd8"}
May 20 14:41:18.958284 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:18.957959    2570 scope.go:117] "RemoveContainer" containerID="5b506e1811924852a113055bdba8d156d075081a9f90d6eaf7f47a925453ca63"
May 20 14:41:18.966399 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:18.966381    2570 scope.go:117] "RemoveContainer" containerID="5b506e1811924852a113055bdba8d156d075081a9f90d6eaf7f47a925453ca63"
May 20 14:41:18.966703 ip-10-0-136-75 kubenswrapper[2570]: E0520 14:41:18.966683    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b506e1811924852a113055bdba8d156d075081a9f90d6eaf7f47a925453ca63\": container with ID starting with 5b506e1811924852a113055bdba8d156d075081a9f90d6eaf7f47a925453ca63 not found: ID does not exist" containerID="5b506e1811924852a113055bdba8d156d075081a9f90d6eaf7f47a925453ca63"
May 20 14:41:18.966763 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:18.966713    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b506e1811924852a113055bdba8d156d075081a9f90d6eaf7f47a925453ca63"} err="failed to get container status \"5b506e1811924852a113055bdba8d156d075081a9f90d6eaf7f47a925453ca63\": rpc error: code = NotFound desc = could not find container \"5b506e1811924852a113055bdba8d156d075081a9f90d6eaf7f47a925453ca63\": container with ID starting with 5b506e1811924852a113055bdba8d156d075081a9f90d6eaf7f47a925453ca63 not found: ID does not exist"
May 20 14:41:18.980637 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:18.980597    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht"]
May 20 14:41:18.984940 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:18.984918    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3e303-66c47b978f-nfvht"]
May 20 14:41:19.363461 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:19.363424    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14cab6c0-a554-416d-ae2e-76b2c089e696" path="/var/lib/kubelet/pods/14cab6c0-a554-416d-ae2e-76b2c089e696/volumes"
May 20 14:41:22.309388 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:22.309358    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kvcpv_a7a2cce0-9de9-4b66-83f3-2c58f1656ed1/global-pull-secret-syncer/0.log"
May 20 14:41:22.477974 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:22.476081    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x6mng_a77563fb-8a39-452d-8c02-dc39cee852cf/konnectivity-agent/0.log"
May 20 14:41:22.531217 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:22.531187    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-75.ec2.internal_6959407f02e6e503e019a9457b3f87d1/haproxy/0.log"
May 20 14:41:26.039280 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:26.039251    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0807f6a1-c649-418f-acde-1ea45476efe2/alertmanager/0.log"
May 20 14:41:26.066433 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:26.066405    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0807f6a1-c649-418f-acde-1ea45476efe2/config-reloader/0.log"
May 20 14:41:26.095152 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:26.095110    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0807f6a1-c649-418f-acde-1ea45476efe2/kube-rbac-proxy-web/0.log"
May 20 14:41:26.122954 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:26.122928    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0807f6a1-c649-418f-acde-1ea45476efe2/kube-rbac-proxy/0.log"
May 20 14:41:26.148532 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:26.148504    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0807f6a1-c649-418f-acde-1ea45476efe2/kube-rbac-proxy-metric/0.log"
May 20 14:41:26.173844 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:26.173811    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0807f6a1-c649-418f-acde-1ea45476efe2/prom-label-proxy/0.log"
May 20 14:41:26.202770 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:26.202750    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0807f6a1-c649-418f-acde-1ea45476efe2/init-config-reloader/0.log"
May 20 14:41:26.356028 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:26.355940    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-jxt2d_c1998859-b3d0-4755-bdb3-cb4ccb164e16/kube-state-metrics/0.log"
May 20 14:41:26.392175 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:26.392146    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-jxt2d_c1998859-b3d0-4755-bdb3-cb4ccb164e16/kube-rbac-proxy-main/0.log"
May 20 14:41:26.422102 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:26.422065    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-jxt2d_c1998859-b3d0-4755-bdb3-cb4ccb164e16/kube-rbac-proxy-self/0.log"
May 20 14:41:26.484173 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:26.484138    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-hnpg6_756758ba-486b-42df-9c73-3f52e08e5da6/monitoring-plugin/0.log"
May 20 14:41:26.614927 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:26.614842    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9cc64_a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed/node-exporter/0.log"
May 20 14:41:26.645591 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:26.645561    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9cc64_a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed/kube-rbac-proxy/0.log"
May 20 14:41:26.671082 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:26.671058    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9cc64_a30b7f91-5c4f-43be-9ae4-b5a03e2e3fed/init-textfile/0.log"
May 20 14:41:27.306092 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:27.306062    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-2qf4d_2935ba18-80fa-4e6f-b4c1-6fe81a9b7981/prometheus-operator/0.log"
May 20 14:41:27.330323 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:27.330249    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-2qf4d_2935ba18-80fa-4e6f-b4c1-6fe81a9b7981/kube-rbac-proxy/0.log"
May 20 14:41:27.397585 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:27.397553    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6494fbb744-z97hh_ad7632bd-8b00-44f6-af8b-66763c3175c7/telemeter-client/0.log"
May 20 14:41:27.425107 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:27.425077    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6494fbb744-z97hh_ad7632bd-8b00-44f6-af8b-66763c3175c7/reload/0.log"
May 20 14:41:27.454868 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:27.454815    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6494fbb744-z97hh_ad7632bd-8b00-44f6-af8b-66763c3175c7/kube-rbac-proxy/0.log"
May 20 14:41:28.976332 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.976297    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg"]
May 20 14:41:28.976743 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.976651    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="247807be-896a-45f3-9ae9-5266e0b9fa63" containerName="kserve-container"
May 20 14:41:28.976743 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.976664    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="247807be-896a-45f3-9ae9-5266e0b9fa63" containerName="kserve-container"
May 20 14:41:28.976743 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.976682    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="247807be-896a-45f3-9ae9-5266e0b9fa63" containerName="kube-rbac-proxy"
May 20 14:41:28.976743 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.976687    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="247807be-896a-45f3-9ae9-5266e0b9fa63" containerName="kube-rbac-proxy"
May 20 14:41:28.976743 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.976695    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="029acfcc-1456-4f49-85a1-a95d8ab677d1" containerName="splitter-graph-31b2e"
May 20 14:41:28.976743 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.976700    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="029acfcc-1456-4f49-85a1-a95d8ab677d1" containerName="splitter-graph-31b2e"
May 20 14:41:28.976743 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.976707    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01a6cb98-37b5-49c2-a601-d09952a1df97" containerName="kserve-container"
May 20 14:41:28.976743 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.976712    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="01a6cb98-37b5-49c2-a601-d09952a1df97" containerName="kserve-container"
May 20 14:41:28.976743 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.976718    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01a6cb98-37b5-49c2-a601-d09952a1df97" containerName="kube-rbac-proxy"
May 20 14:41:28.976743 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.976724    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="01a6cb98-37b5-49c2-a601-d09952a1df97" containerName="kube-rbac-proxy"
May 20 14:41:28.976743 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.976732    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14cab6c0-a554-416d-ae2e-76b2c089e696" containerName="switch-graph-3e303"
May 20 14:41:28.976743 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.976738    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="14cab6c0-a554-416d-ae2e-76b2c089e696" containerName="switch-graph-3e303"
May 20 14:41:28.977158 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.976785    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="01a6cb98-37b5-49c2-a601-d09952a1df97" containerName="kube-rbac-proxy"
May 20 14:41:28.977158 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.976792    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="029acfcc-1456-4f49-85a1-a95d8ab677d1" containerName="splitter-graph-31b2e"
May 20 14:41:28.977158 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.976800    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="01a6cb98-37b5-49c2-a601-d09952a1df97" containerName="kserve-container"
May 20 14:41:28.977158 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.976806    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="247807be-896a-45f3-9ae9-5266e0b9fa63" containerName="kube-rbac-proxy"
May 20 14:41:28.977158 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.976814    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="247807be-896a-45f3-9ae9-5266e0b9fa63" containerName="kserve-container"
May 20 14:41:28.977158 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.976820    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="14cab6c0-a554-416d-ae2e-76b2c089e696" containerName="switch-graph-3e303"
May 20 14:41:28.981193 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.981169    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg"
May 20 14:41:28.983780 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.983755    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lvfml\"/\"kube-root-ca.crt\""
May 20 14:41:28.983907 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.983802    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lvfml\"/\"openshift-service-ca.crt\""
May 20 14:41:28.984750 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.984736    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lvfml\"/\"default-dockercfg-54sx6\""
May 20 14:41:28.989257 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:28.989186    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg"]
May 20 14:41:29.033880 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:29.033845    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f820a352-71eb-45e2-8fdc-64e7c605e947-lib-modules\") pod \"perf-node-gather-daemonset-k4kgg\" (UID: \"f820a352-71eb-45e2-8fdc-64e7c605e947\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg"
May 20 14:41:29.034045 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:29.033914    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdrm9\" (UniqueName: \"kubernetes.io/projected/f820a352-71eb-45e2-8fdc-64e7c605e947-kube-api-access-hdrm9\") pod \"perf-node-gather-daemonset-k4kgg\" (UID: \"f820a352-71eb-45e2-8fdc-64e7c605e947\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg"
May 20 14:41:29.034045 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:29.034007    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f820a352-71eb-45e2-8fdc-64e7c605e947-proc\") pod \"perf-node-gather-daemonset-k4kgg\" (UID: \"f820a352-71eb-45e2-8fdc-64e7c605e947\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg"
May 20 14:41:29.034045 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:29.034037    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f820a352-71eb-45e2-8fdc-64e7c605e947-sys\") pod \"perf-node-gather-daemonset-k4kgg\" (UID: \"f820a352-71eb-45e2-8fdc-64e7c605e947\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg"
May 20 14:41:29.034175 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:29.034065    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f820a352-71eb-45e2-8fdc-64e7c605e947-podres\") pod \"perf-node-gather-daemonset-k4kgg\" (UID: \"f820a352-71eb-45e2-8fdc-64e7c605e947\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg"
May 20 14:41:29.134766 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:29.134725    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f820a352-71eb-45e2-8fdc-64e7c605e947-proc\") pod \"perf-node-gather-daemonset-k4kgg\" (UID: \"f820a352-71eb-45e2-8fdc-64e7c605e947\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg"
May 20 14:41:29.134766 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:29.134763    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f820a352-71eb-45e2-8fdc-64e7c605e947-sys\") pod \"perf-node-gather-daemonset-k4kgg\" (UID: \"f820a352-71eb-45e2-8fdc-64e7c605e947\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg"
May 20 14:41:29.134972 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:29.134789    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f820a352-71eb-45e2-8fdc-64e7c605e947-podres\") pod \"perf-node-gather-daemonset-k4kgg\" (UID: \"f820a352-71eb-45e2-8fdc-64e7c605e947\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg"
May 20 14:41:29.134972 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:29.134816    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f820a352-71eb-45e2-8fdc-64e7c605e947-lib-modules\") pod \"perf-node-gather-daemonset-k4kgg\" (UID: \"f820a352-71eb-45e2-8fdc-64e7c605e947\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg"
May 20 14:41:29.134972 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:29.134872    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hdrm9\" (UniqueName: \"kubernetes.io/projected/f820a352-71eb-45e2-8fdc-64e7c605e947-kube-api-access-hdrm9\") pod \"perf-node-gather-daemonset-k4kgg\" (UID: \"f820a352-71eb-45e2-8fdc-64e7c605e947\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg"
May 20 14:41:29.134972 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:29.134870    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f820a352-71eb-45e2-8fdc-64e7c605e947-sys\") pod \"perf-node-gather-daemonset-k4kgg\" (UID: \"f820a352-71eb-45e2-8fdc-64e7c605e947\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg"
May 20 14:41:29.134972 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:29.134870    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f820a352-71eb-45e2-8fdc-64e7c605e947-proc\") pod \"perf-node-gather-daemonset-k4kgg\" (UID: \"f820a352-71eb-45e2-8fdc-64e7c605e947\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg"
May 20 14:41:29.134972 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:29.134938    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f820a352-71eb-45e2-8fdc-64e7c605e947-podres\") pod \"perf-node-gather-daemonset-k4kgg\" (UID: \"f820a352-71eb-45e2-8fdc-64e7c605e947\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg"
May 20 14:41:29.134972 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:29.134963    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f820a352-71eb-45e2-8fdc-64e7c605e947-lib-modules\") pod \"perf-node-gather-daemonset-k4kgg\" (UID: \"f820a352-71eb-45e2-8fdc-64e7c605e947\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg"
May 20 14:41:29.144514 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:29.144479    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdrm9\" (UniqueName: \"kubernetes.io/projected/f820a352-71eb-45e2-8fdc-64e7c605e947-kube-api-access-hdrm9\") pod \"perf-node-gather-daemonset-k4kgg\" (UID: \"f820a352-71eb-45e2-8fdc-64e7c605e947\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg"
May 20 14:41:29.292796 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:29.292767    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg"
May 20 14:41:29.427894 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:29.427870    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg"]
May 20 14:41:29.430046 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:29.430023    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5f5b9cdfc5-hh7lz_f69e3276-e4a9-47fa-a1db-e18f14eba62b/console/0.log"
May 20 14:41:29.430577 ip-10-0-136-75 kubenswrapper[2570]: W0520 14:41:29.430547    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf820a352_71eb_45e2_8fdc_64e7c605e947.slice/crio-37119e55d5ec643dcc9912b7ecdddf9bedbc368b7dbc6cd09ae558924c0bfeb1 WatchSource:0}: Error finding container 37119e55d5ec643dcc9912b7ecdddf9bedbc368b7dbc6cd09ae558924c0bfeb1: Status 404 returned error can't find the container with id 37119e55d5ec643dcc9912b7ecdddf9bedbc368b7dbc6cd09ae558924c0bfeb1
May 20 14:41:29.432006 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:29.431992    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:41:29.996176 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:29.996136    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg" event={"ID":"f820a352-71eb-45e2-8fdc-64e7c605e947","Type":"ContainerStarted","Data":"a71ad52965ad3061eb3c7cdf24b23bfbcfc6e5c126b2565f817afe27f2871862"}
May 20 14:41:29.996176 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:29.996180    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg" event={"ID":"f820a352-71eb-45e2-8fdc-64e7c605e947","Type":"ContainerStarted","Data":"37119e55d5ec643dcc9912b7ecdddf9bedbc368b7dbc6cd09ae558924c0bfeb1"}
May 20 14:41:29.996602 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:29.996210    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg"
May 20 14:41:30.016417 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:30.016369    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg" podStartSLOduration=2.016355823 podStartE2EDuration="2.016355823s" podCreationTimestamp="2026-05-20 14:41:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:41:30.013791031 +0000 UTC m=+3057.251407679" watchObservedRunningTime="2026-05-20 14:41:30.016355823 +0000 UTC m=+3057.253972469"
May 20 14:41:30.613924 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:30.613899    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-df47p_0bb7fe7c-d50a-4d19-8be0-ded9425983e3/dns/0.log"
May 20 14:41:30.638716 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:30.638688    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-df47p_0bb7fe7c-d50a-4d19-8be0-ded9425983e3/kube-rbac-proxy/0.log"
May 20 14:41:30.716508 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:30.716476    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cs8gn_b985f05d-7aa5-42b7-a5d7-3b18a2769356/dns-node-resolver/0.log"
May 20 14:41:31.255948 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:31.255914    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dk92v_efd2c3ea-0867-4025-b5c6-03b9a0d744c5/node-ca/0.log"
May 20 14:41:32.410630 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:32.410586    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-np4p2_030d660b-7832-4b43-bf8f-d280dc34a607/serve-healthcheck-canary/0.log"
May 20 14:41:32.992699 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:32.992665    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n9979_e95098e3-08f2-4619-8504-9cdc39ae3b80/kube-rbac-proxy/0.log"
May 20 14:41:33.019744 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:33.019715    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n9979_e95098e3-08f2-4619-8504-9cdc39ae3b80/exporter/0.log"
May 20 14:41:33.046265 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:33.046237    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n9979_e95098e3-08f2-4619-8504-9cdc39ae3b80/extractor/0.log"
May 20 14:41:35.071010 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:35.070977    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-867cffc775-hpvm4_1125b1e5-8fbf-40e8-872b-61ce1ce07ec3/manager/0.log"
May 20 14:41:35.094316 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:35.094291    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-68cc5db7c4-gh9gs_2503b4b5-50c4-40d2-8e74-6de98c022740/manager/0.log"
May 20 14:41:35.120669 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:35.120641    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-q46ld_5f907942-12ba-46b9-979a-90c37b400ca0/server/0.log"
May 20 14:41:35.361935 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:35.361845    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-cp94v_fef87a76-8df3-43e2-af68-0ec0f069a524/manager/0.log"
May 20 14:41:35.450196 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:35.450170    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-79f457656b-dt4x8_2df218e5-e163-4608-9c2a-b80f6f2b82a9/seaweedfs/0.log"
May 20 14:41:36.010217 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:36.010191    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-k4kgg"
May 20 14:41:41.326886 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:41.326851    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9wtjr_cc57a4cf-cc2d-43b4-a0a4-de6a545405a3/kube-multus/0.log"
May 20 14:41:41.744838 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:41.744757    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zgj4t_ead02d78-f55b-4612-bb8d-057d70f6ecc4/kube-multus-additional-cni-plugins/0.log"
May 20 14:41:41.776648 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:41.776581    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zgj4t_ead02d78-f55b-4612-bb8d-057d70f6ecc4/egress-router-binary-copy/0.log"
May 20 14:41:41.803499 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:41.803464    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zgj4t_ead02d78-f55b-4612-bb8d-057d70f6ecc4/cni-plugins/0.log"
May 20 14:41:41.831038 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:41.831007    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zgj4t_ead02d78-f55b-4612-bb8d-057d70f6ecc4/bond-cni-plugin/0.log"
May 20 14:41:41.863513 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:41.863478    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zgj4t_ead02d78-f55b-4612-bb8d-057d70f6ecc4/routeoverride-cni/0.log"
May 20 14:41:41.892458 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:41.892432    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zgj4t_ead02d78-f55b-4612-bb8d-057d70f6ecc4/whereabouts-cni-bincopy/0.log"
May 20 14:41:41.919791 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:41.919763    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zgj4t_ead02d78-f55b-4612-bb8d-057d70f6ecc4/whereabouts-cni/0.log"
May 20 14:41:42.104712 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:42.104687    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rhcx7_6969002d-0b97-4cb6-ae5b-03d5ef9dddf8/network-metrics-daemon/0.log"
May 20 14:41:42.131696 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:42.131674    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rhcx7_6969002d-0b97-4cb6-ae5b-03d5ef9dddf8/kube-rbac-proxy/0.log"
May 20 14:41:42.938334 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:42.938305    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-controller/0.log"
May 20 14:41:42.972654 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:42.972604    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/0.log"
May 20 14:41:42.988669 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:42.988642    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovn-acl-logging/1.log"
May 20 14:41:43.008818 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:43.008783    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/kube-rbac-proxy-node/0.log"
May 20 14:41:43.034175 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:43.034148    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/kube-rbac-proxy-ovn-metrics/0.log"
May 20 14:41:43.066447 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:43.066418    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/northd/0.log"
May 20 14:41:43.093751 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:43.093729    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/nbdb/0.log"
May 20 14:41:43.121115 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:43.121087    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/sbdb/0.log"
May 20 14:41:43.232303 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:43.232224    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfhg5_56992f6a-130b-435f-a75b-bf6b103ec8e1/ovnkube-controller/0.log"
May 20 14:41:44.880101 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:44.880072    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-clw2n_31842022-6a01-49b0-8b26-fb0d882cdf6f/network-check-target-container/0.log"
May 20 14:41:45.975880 ip-10-0-136-75 kubenswrapper[2570]: I0520 14:41:45.975844    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-bwkmk_3142b56b-8ab7-4799-8b7a-46420a701680/iptables-alerter/0.log"