May 20 15:28:59.035764 ip-10-0-129-117 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 20 15:28:59.035777 ip-10-0-129-117 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 20 15:28:59.035785 ip-10-0-129-117 systemd[1]: kubelet.service: Failed with result 'resources'.
May 20 15:28:59.036106 ip-10-0-129-117 systemd[1]: Failed to start Kubernetes Kubelet.
May 20 15:29:10.260206 ip-10-0-129-117 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 20 15:29:10.260228 ip-10-0-129-117 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d8359a4664a443b6babc60171c116139 --
May 20 15:31:30.933340 ip-10-0-129-117 systemd[1]: Starting Kubernetes Kubelet...
May 20 15:31:31.417592 ip-10-0-129-117 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 15:31:31.417592 ip-10-0-129-117 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 20 15:31:31.417592 ip-10-0-129-117 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 15:31:31.417592 ip-10-0-129-117 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 20 15:31:31.417592 ip-10-0-129-117 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 15:31:31.421865 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.421777    2568 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 20 15:31:31.429263 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429242    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:31:31.429263 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429259    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:31:31.429327 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429266    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:31:31.429327 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429270    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:31:31.429327 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429272    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:31:31.429327 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429275    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:31:31.429327 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429278    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:31:31.429327 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429281    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:31:31.429327 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429289    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:31:31.429327 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429292    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:31:31.429327 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429295    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:31:31.429327 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429297    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:31:31.429327 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429300    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:31:31.429327 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429302    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:31:31.429327 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429305    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:31:31.429327 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429307    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:31:31.429327 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429310    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:31:31.429327 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429312    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:31:31.429327 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429315    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:31:31.429327 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429318    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:31:31.429327 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429321    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:31:31.429327 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429323    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:31:31.429791 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429326    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:31:31.429791 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429328    2568 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:31:31.429791 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429331    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:31:31.429791 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429334    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:31:31.429791 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429337    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:31:31.429791 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429339    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:31:31.429791 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429342    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:31:31.429791 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429344    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:31:31.429791 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429347    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:31:31.429791 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429349    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:31:31.429791 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429352    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:31:31.429791 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429354    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:31:31.429791 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429356    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:31:31.429791 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429359    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:31:31.429791 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429362    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:31:31.429791 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429364    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:31:31.429791 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429367    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:31:31.429791 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429369    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:31:31.429791 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429371    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:31:31.429791 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429374    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:31:31.430354 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429376    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:31:31.430354 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429378    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:31:31.430354 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429381    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:31:31.430354 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429383    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:31:31.430354 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429386    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:31:31.430354 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429389    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:31:31.430354 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429392    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:31:31.430354 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429395    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:31:31.430354 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429397    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:31:31.430354 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429400    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:31:31.430354 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429402    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:31:31.430354 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429405    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:31:31.430354 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429408    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:31:31.430354 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429410    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:31:31.430354 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429412    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:31:31.430354 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429415    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:31:31.430354 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429417    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:31:31.430354 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429420    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:31:31.430354 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429422    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:31:31.430354 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429425    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:31:31.430832 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429427    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:31:31.430832 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429432    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:31:31.430832 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429435    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:31:31.430832 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429438    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:31:31.430832 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429440    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:31:31.430832 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429443    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:31:31.430832 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429445    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:31:31.430832 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429451    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:31:31.430832 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429453    2568 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:31:31.430832 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429458    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:31:31.430832 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429462    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:31:31.430832 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429465    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:31:31.430832 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429467    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:31:31.430832 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429470    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:31:31.430832 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429472    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:31:31.430832 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429475    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:31:31.430832 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429478    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:31:31.430832 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429480    2568 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:31:31.430832 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429483    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:31:31.431309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429486    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:31:31.431309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429488    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:31:31.431309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429491    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:31:31.431309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429493    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:31:31.431309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429496    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:31:31.431309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429901    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:31:31.431309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429907    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:31:31.431309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429909    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:31:31.431309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429912    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:31:31.431309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429915    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:31:31.431309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429918    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:31:31.431309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429921    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:31:31.431309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429923    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:31:31.431309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429926    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:31:31.431309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429928    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:31:31.431309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429931    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:31:31.431309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429933    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:31:31.431309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429935    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:31:31.431309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429938    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:31:31.431309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429940    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:31:31.431820 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429943    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:31:31.431820 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429946    2568 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:31:31.431820 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429949    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:31:31.431820 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429951    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:31:31.431820 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429954    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:31:31.431820 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429956    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:31:31.431820 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429960    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:31:31.431820 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429963    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:31:31.431820 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429966    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:31:31.431820 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429969    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:31:31.431820 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429972    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:31:31.431820 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429974    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:31:31.431820 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429977    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:31:31.431820 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429979    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:31:31.431820 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429982    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:31:31.431820 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429985    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:31:31.431820 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429987    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:31:31.431820 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429991    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:31:31.431820 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429993    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:31:31.432309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429996    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:31:31.432309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.429998    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:31:31.432309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430000    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:31:31.432309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430003    2568 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:31:31.432309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430006    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:31:31.432309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430009    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:31:31.432309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430012    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:31:31.432309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430014    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:31:31.432309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430037    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:31:31.432309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430041    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:31:31.432309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430045    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:31:31.432309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430050    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:31:31.432309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430052    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:31:31.432309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430055    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:31:31.432309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430058    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:31:31.432309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430061    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:31:31.432309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430063    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:31:31.432309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430066    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:31:31.432309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430068    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:31:31.432309 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430071    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:31:31.432807 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430073    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:31:31.432807 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430077    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:31:31.432807 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430079    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:31:31.432807 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430082    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:31:31.432807 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430084    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:31:31.432807 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430087    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:31:31.432807 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430090    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:31:31.432807 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430093    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:31:31.432807 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430096    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:31:31.432807 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430099    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:31:31.432807 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430101    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:31:31.432807 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430104    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:31:31.432807 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430107    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:31:31.432807 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430110    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:31:31.432807 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430113    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:31:31.432807 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430115    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:31:31.432807 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430117    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:31:31.432807 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430120    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:31:31.432807 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430123    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:31:31.432807 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430125    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:31:31.433318 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430127    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:31:31.433318 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430130    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:31:31.433318 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430132    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:31:31.433318 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430136    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:31:31.433318 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430138    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:31:31.433318 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430141    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:31:31.433318 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430143    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:31:31.433318 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430146    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:31:31.433318 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430149    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:31:31.433318 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430151    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:31:31.433318 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430154    2568 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:31:31.433318 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.430157    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:31:31.433318 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430869    2568 flags.go:64] FLAG: --address="0.0.0.0"
May 20 15:31:31.433318 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430879    2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 20 15:31:31.433318 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430885    2568 flags.go:64] FLAG: --anonymous-auth="true"
May 20 15:31:31.433318 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430890    2568 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 20 15:31:31.433318 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430895    2568 flags.go:64] FLAG: --authentication-token-webhook="false"
May 20 15:31:31.433318 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430899    2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 20 15:31:31.433318 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430903    2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 20 15:31:31.433318 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430907    2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 20 15:31:31.433318 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430910    2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430913    2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430917    2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430920    2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430923    2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430926    2568 flags.go:64] FLAG: --cgroup-root=""
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430929    2568 flags.go:64] FLAG: --cgroups-per-qos="true"
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430932    2568 flags.go:64] FLAG: --client-ca-file=""
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430935    2568 flags.go:64] FLAG: --cloud-config=""
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430937    2568 flags.go:64] FLAG: --cloud-provider="external"
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430940    2568 flags.go:64] FLAG: --cluster-dns="[]"
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430944    2568 flags.go:64] FLAG: --cluster-domain=""
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430947    2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430951    2568 flags.go:64] FLAG: --config-dir=""
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430953    2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430957    2568 flags.go:64] FLAG: --container-log-max-files="5"
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430960    2568 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430963    2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430967    2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430970    2568 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430973    2568 flags.go:64] FLAG: --contention-profiling="false"
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430976    2568 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430980    2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430983    2568 flags.go:64] FLAG: --cpu-manager-policy="none"
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430986    2568 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 20 15:31:31.433826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430990    2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430992    2568 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430995    2568 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.430998    2568 flags.go:64] FLAG: --enable-load-reader="false"
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431001    2568 flags.go:64] FLAG: --enable-server="true"
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431004    2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431009    2568 flags.go:64] FLAG: --event-burst="100"
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431012    2568 flags.go:64] FLAG: --event-qps="50"
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431033    2568 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431037    2568 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431040    2568 flags.go:64] FLAG: --eviction-hard=""
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431044    2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431046    2568 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431055    2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431059    2568 flags.go:64] FLAG: --eviction-soft=""
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431062    2568 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431064    2568 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431067    2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431070    2568 flags.go:64] FLAG: --experimental-mounter-path=""
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431073    2568 flags.go:64] FLAG: --fail-cgroupv1="false"
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431076    2568 flags.go:64] FLAG: --fail-swap-on="true"
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431079    2568 flags.go:64] FLAG: --feature-gates=""
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431083    2568 flags.go:64] FLAG: --file-check-frequency="20s"
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431087    2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431090    2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 20 15:31:31.434428 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431093    2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431096    2568 flags.go:64] FLAG: --healthz-port="10248"
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431099    2568 flags.go:64] FLAG: --help="false"
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431107    2568 flags.go:64] FLAG: --hostname-override="ip-10-0-129-117.ec2.internal"
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431110    2568 flags.go:64] FLAG: --housekeeping-interval="10s"
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431114    2568 flags.go:64] FLAG: --http-check-frequency="20s"
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431116    2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431120    2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431123    2568 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431126    2568 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431129    2568 flags.go:64] FLAG: --image-service-endpoint=""
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431131    2568 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431134    2568 flags.go:64] FLAG: --kube-api-burst="100"
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431137    2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431141    2568 flags.go:64] FLAG: --kube-api-qps="50"
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431145    2568 flags.go:64] FLAG: --kube-reserved=""
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431147    2568 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431150    2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431153    2568 flags.go:64] FLAG: --kubelet-cgroups=""
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431156    2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431159    2568 flags.go:64] FLAG: --lock-file=""
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431162    2568 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431165    2568 flags.go:64] FLAG: --log-flush-frequency="5s"
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431168    2568 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 20 15:31:31.435081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431173    2568 flags.go:64] FLAG: --log-json-split-stream="false"
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431176    2568 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431179    2568 flags.go:64] FLAG: --log-text-split-stream="false"
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431182    2568 flags.go:64] FLAG: --logging-format="text"
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431187    2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431191    2568 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431193    2568 flags.go:64] FLAG: --manifest-url=""
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431196    2568 flags.go:64] FLAG: --manifest-url-header=""
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431200    2568 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431203    2568 flags.go:64] FLAG: --max-open-files="1000000"
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431207    2568 flags.go:64] FLAG: --max-pods="110"
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431210    2568 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431221    2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431224    2568 flags.go:64] FLAG: --memory-manager-policy="None"
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431227    2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431229    2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431232    2568 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431235    2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431244    2568 flags.go:64] FLAG: --node-status-max-images="50"
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431247    2568 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431251    2568 flags.go:64] FLAG: --oom-score-adj="-999"
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431254    2568 flags.go:64] FLAG: --pod-cidr=""
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431257    2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 20 15:31:31.435651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431263    2568 flags.go:64] FLAG: --pod-manifest-path=""
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431266    2568 flags.go:64] FLAG: --pod-max-pids="-1"
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431269    2568 flags.go:64] FLAG: --pods-per-core="0"
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431272    2568 flags.go:64] FLAG: --port="10250"
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431275    2568 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431278    2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0629138be6851ea4f"
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431281    2568 flags.go:64] FLAG: --qos-reserved=""
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431284    2568 flags.go:64] FLAG: --read-only-port="10255"
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431287    2568 flags.go:64] FLAG: --register-node="true"
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431290    2568 flags.go:64] FLAG: --register-schedulable="true"
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431292    2568 flags.go:64] FLAG: --register-with-taints=""
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431296    2568 flags.go:64] FLAG: --registry-burst="10"
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431299    2568 flags.go:64] FLAG: --registry-qps="5"
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431302    2568 flags.go:64] FLAG: --reserved-cpus=""
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431305    2568 flags.go:64] FLAG: --reserved-memory=""
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431309    2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431312    2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431315    2568 flags.go:64] FLAG: --rotate-certificates="false"
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431318    2568 flags.go:64] FLAG: --rotate-server-certificates="false"
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431321    2568 flags.go:64] FLAG: --runonce="false"
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431324    2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431327    2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431333    2568 flags.go:64] FLAG: --seccomp-default="false"
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431336    2568 flags.go:64] FLAG: --serialize-image-pulls="true"
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431339    2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431342    2568 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 20 15:31:31.436234 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431345    2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431348    2568 flags.go:64] FLAG: --storage-driver-password="root"
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431352    2568 flags.go:64] FLAG: --storage-driver-secure="false"
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431355    2568 flags.go:64] FLAG: --storage-driver-table="stats"
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431361    2568 flags.go:64] FLAG: --storage-driver-user="root"
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431364    2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431367    2568 flags.go:64] FLAG: --sync-frequency="1m0s"
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431370    2568 flags.go:64] FLAG: --system-cgroups=""
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431373    2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431378    2568 flags.go:64] FLAG: --system-reserved-cgroup=""
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431381    2568 flags.go:64] FLAG: --tls-cert-file=""
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431384    2568 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431387    2568 flags.go:64] FLAG: --tls-min-version=""
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431391    2568 flags.go:64] FLAG: --tls-private-key-file=""
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431393    2568 flags.go:64] FLAG: --topology-manager-policy="none"
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431396    2568 flags.go:64] FLAG: --topology-manager-policy-options=""
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431399    2568 flags.go:64] FLAG: --topology-manager-scope="container"
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431402    2568 flags.go:64] FLAG: --v="2"
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431406    2568 flags.go:64] FLAG: --version="false"
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431411    2568 flags.go:64] FLAG: --vmodule=""
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431416    2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.431419    2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431512    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431515    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431518    2568 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:31:31.436872 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431521    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:31:31.437499 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431524    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:31:31.437499 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431526    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:31:31.437499 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431529    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:31:31.437499 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431532    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:31:31.437499 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431534    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:31:31.437499 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431537    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:31:31.437499 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431540    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:31:31.437499 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431542    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:31:31.437499 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431545    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:31:31.437499 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431547    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:31:31.437499 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431551    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:31:31.437499 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431554    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:31:31.437499 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431556    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:31:31.437499 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431558    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:31:31.437499 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431561    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:31:31.437499 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431564    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:31:31.437499 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431566    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:31:31.437499 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431569    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:31:31.437499 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431571    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:31:31.437499 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431573    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:31:31.438003 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431576    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:31:31.438003 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431578    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:31:31.438003 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431581    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:31:31.438003 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431583    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:31:31.438003 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431585    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:31:31.438003 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431589    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:31:31.438003 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431592    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:31:31.438003 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431595    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:31:31.438003 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431597    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:31:31.438003 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431600    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:31:31.438003 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431603    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:31:31.438003 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431606    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:31:31.438003 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431608    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:31:31.438003 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431611    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:31:31.438003 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431613    2568 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:31:31.438003 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431615    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:31:31.438003 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431618    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:31:31.438003 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431621    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:31:31.438003 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431624    2568 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:31:31.438003 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431626    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:31:31.438793 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431628    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:31:31.438793 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431631    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:31:31.438793 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431634    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:31:31.438793 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431637    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:31:31.438793 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431641    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:31:31.438793 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431644    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:31:31.438793 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431647    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:31:31.438793 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431649    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:31:31.438793 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431651    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:31:31.438793 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431654    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:31:31.438793 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431658    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:31:31.438793 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431661    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:31:31.438793 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431665    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:31:31.438793 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431668    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:31:31.438793 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431671    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:31:31.438793 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431673    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:31:31.438793 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431676    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:31:31.438793 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431680    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:31:31.438793 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431683    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:31:31.439617 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431685    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:31:31.439617 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431688    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:31:31.439617 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431690    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:31:31.439617 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431693    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:31:31.439617 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431695    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:31:31.439617 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431698    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:31:31.439617 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431701    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:31:31.439617 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431704    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:31:31.439617 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431706    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:31:31.439617 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431709    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:31:31.439617 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431712    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:31:31.439617 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431714    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:31:31.439617 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431716    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:31:31.439617 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431719    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:31:31.439617 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431721    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:31:31.439617 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431725    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:31:31.439617 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431727    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:31:31.439617 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431730    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:31:31.439617 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431732    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:31:31.439617 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431735    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:31:31.440474 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431738    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:31:31.440474 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431740    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:31:31.440474 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.431743    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:31:31.440474 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.432342    2568 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 15:31:31.441577 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.441555    2568 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 20 15:31:31.441620 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.441580    2568 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 20 15:31:31.441653 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441637    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:31:31.441653 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441643    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:31:31.441653 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441647    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:31:31.441653 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441651    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:31:31.441653 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441654    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:31:31.441773 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441658    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:31:31.441773 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441661    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:31:31.441773 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441664    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:31:31.441773 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441668    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:31:31.441773 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441673    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:31:31.441773 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441677    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:31:31.441773 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441680    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:31:31.441773 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441683    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:31:31.441773 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441686    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:31:31.441773 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441689    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:31:31.441773 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441691    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:31:31.441773 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441694    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:31:31.441773 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441696    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:31:31.441773 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441699    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:31:31.441773 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441702    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:31:31.441773 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441704    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:31:31.441773 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441707    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:31:31.441773 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441709    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:31:31.441773 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441712    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:31:31.441773 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441714    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:31:31.442314 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441717    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:31:31.442314 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441720    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:31:31.442314 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441723    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:31:31.442314 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441725    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:31:31.442314 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441728    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:31:31.442314 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441731    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:31:31.442314 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441734    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:31:31.442314 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441736    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:31:31.442314 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441739    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:31:31.442314 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441741    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:31:31.442314 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441744    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:31:31.442314 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441746    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:31:31.442314 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441749    2568 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:31:31.442314 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441752    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:31:31.442314 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441755    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:31:31.442314 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441757    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:31:31.442314 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441760    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:31:31.442314 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441762    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:31:31.442314 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441765    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:31:31.442314 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441768    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:31:31.442806 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441770    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:31:31.442806 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441773    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:31:31.442806 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441775    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:31:31.442806 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441778    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:31:31.442806 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441780    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:31:31.442806 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441783    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:31:31.442806 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441786    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:31:31.442806 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441788    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:31:31.442806 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441791    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:31:31.442806 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441793    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:31:31.442806 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441797    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:31:31.442806 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441801    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:31:31.442806 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441804    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:31:31.442806 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441807    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:31:31.442806 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441810    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:31:31.442806 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441813    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:31:31.442806 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441815    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:31:31.442806 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441818    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:31:31.442806 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441820    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:31:31.443287 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441823    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:31:31.443287 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441825    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:31:31.443287 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441828    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:31:31.443287 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441831    2568 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:31:31.443287 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441833    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:31:31.443287 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441835    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:31:31.443287 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441839    2568 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:31:31.443287 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441842    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:31:31.443287 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441845    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:31:31.443287 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441847    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:31:31.443287 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441850    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:31:31.443287 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441853    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:31:31.443287 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441855    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:31:31.443287 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441858    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:31:31.443287 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441860    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:31:31.443287 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441863    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:31:31.443287 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441865    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:31:31.443287 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441868    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:31:31.443287 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441871    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:31:31.443287 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441873    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:31:31.443774 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441875    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:31:31.443774 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441878    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:31:31.443774 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.441883    2568 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 15:31:31.443774 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441978    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:31:31.443774 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441983    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:31:31.443774 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441986    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:31:31.443774 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441989    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:31:31.443774 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441992    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:31:31.443774 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441995    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:31:31.443774 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.441999    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:31:31.443774 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442001    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:31:31.443774 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442004    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:31:31.443774 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442006    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:31:31.443774 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442009    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:31:31.443774 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442011    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:31:31.444160 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442014    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:31:31.444160 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442033    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:31:31.444160 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442038    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:31:31.444160 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442042    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:31:31.444160 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442045    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:31:31.444160 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442047    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:31:31.444160 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442050    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:31:31.444160 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442053    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:31:31.444160 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442055    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:31:31.444160 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442058    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:31:31.444160 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442070    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:31:31.444160 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442073    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:31:31.444160 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442076    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:31:31.444160 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442078    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:31:31.444160 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442081    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:31:31.444160 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442083    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:31:31.444160 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442086    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:31:31.444160 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442089    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:31:31.444160 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442091    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:31:31.444160 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442094    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:31:31.444646 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442097    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:31:31.444646 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442100    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:31:31.444646 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442102    2568 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:31:31.444646 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442105    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:31:31.444646 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442107    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:31:31.444646 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442111    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:31:31.444646 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442115    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:31:31.444646 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442117    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:31:31.444646 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442120    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:31:31.444646 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442123    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:31:31.444646 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442126    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:31:31.444646 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442129    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:31:31.444646 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442132    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:31:31.444646 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442134    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:31:31.444646 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442137    2568 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:31:31.444646 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442140    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:31:31.444646 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442142    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:31:31.444646 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442145    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:31:31.444646 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442147    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:31:31.445148 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442149    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:31:31.445148 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442152    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:31:31.445148 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442155    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:31:31.445148 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442157    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:31:31.445148 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442161    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:31:31.445148 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442164    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:31:31.445148 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442166    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:31:31.445148 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442169    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:31:31.445148 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442172    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:31:31.445148 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442174    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:31:31.445148 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442177    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:31:31.445148 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442180    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:31:31.445148 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442183    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:31:31.445148 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442185    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:31:31.445148 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442188    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:31:31.445148 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442190    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:31:31.445148 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442193    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:31:31.445148 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442195    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:31:31.445148 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442198    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:31:31.445148 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442201    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:31:31.445670 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442203    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:31:31.445670 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442205    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:31:31.445670 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442208    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:31:31.445670 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442211    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:31:31.445670 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442213    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:31:31.445670 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442216    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:31:31.445670 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442218    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:31:31.445670 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442221    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:31:31.445670 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442223    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:31:31.445670 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442226    2568 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:31:31.445670 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442228    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:31:31.445670 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442231    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:31:31.445670 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442233    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:31:31.445670 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442235    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:31:31.445670 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:31.442238    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:31:31.445670 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.442243    2568 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 15:31:31.446086 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.443045    2568 server.go:962] "Client rotation is on, will bootstrap in background"
May 20 15:31:31.446155 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.446141    2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 20 15:31:31.447215 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.447203    2568 server.go:1019] "Starting client certificate rotation"
May 20 15:31:31.447319 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.447300    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 20 15:31:31.447364 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.447339    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 20 15:31:31.473797 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.473770    2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 20 15:31:31.475940 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.475917    2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 20 15:31:31.486767 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.486741    2568 log.go:25] "Validated CRI v1 runtime API"
May 20 15:31:31.493175 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.493159    2568 log.go:25] "Validated CRI v1 image API"
May 20 15:31:31.495814 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.495786    2568 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 20 15:31:31.498756 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.498725    2568 fs.go:135] Filesystem UUIDs: map[00d68fbe-ec05-4d5a-9a4b-70e6255363a0:/dev/nvme0n1p3 5d6e38ce-0f93-4563-b789-cb802e06eb14:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
May 20 15:31:31.498845 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.498754    2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 20 15:31:31.501999 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.501975    2568 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 20 15:31:31.504402 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.504288    2568 manager.go:217] Machine: {Timestamp:2026-05-20 15:31:31.50377418 +0000 UTC m=+0.442615727 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110213 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b6d531934428dc867c534af3d7433 SystemUUID:ec2b6d53-1934-428d-c867-c534af3d7433 BootID:d8359a46-64a4-43b6-babc-60171c116139 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:38:56:ba:76:43 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:38:56:ba:76:43 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:5b:b1:ca:02:df Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 20 15:31:31.505119 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.505107    2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 20 15:31:31.505213 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.505200    2568 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 20 15:31:31.507871 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.507845    2568 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 20 15:31:31.508010 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.507874    2568 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-117.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 20 15:31:31.508080 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.508038    2568 topology_manager.go:138] "Creating topology manager with none policy"
May 20 15:31:31.508080 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.508048    2568 container_manager_linux.go:306] "Creating device plugin manager"
May 20 15:31:31.508080 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.508056    2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 20 15:31:31.508806 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.508796    2568 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 20 15:31:31.510623 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.510612    2568 state_mem.go:36] "Initialized new in-memory state store"
May 20 15:31:31.510731 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.510722    2568 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 20 15:31:31.513818 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.513808    2568 kubelet.go:491] "Attempting to sync node with API server"
May 20 15:31:31.513862 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.513822    2568 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 20 15:31:31.513862 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.513839    2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 20 15:31:31.513862 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.513848    2568 kubelet.go:397] "Adding apiserver pod source"
May 20 15:31:31.513862 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.513857    2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 20 15:31:31.515149 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.515135    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 20 15:31:31.515199 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.515164    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 20 15:31:31.521452 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.521434    2568 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 20 15:31:31.523448 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.523433    2568 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 20 15:31:31.524264 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.524253    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 20 15:31:31.524326 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.524271    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 20 15:31:31.524326 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.524277    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 20 15:31:31.524326 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.524283    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 20 15:31:31.524326 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.524295    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 20 15:31:31.524326 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.524305    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 20 15:31:31.524326 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.524311    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 20 15:31:31.524326 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.524316    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 20 15:31:31.524326 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.524324    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 20 15:31:31.524326 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.524329    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 20 15:31:31.524597 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.524338    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 20 15:31:31.524628 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.524608    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 20 15:31:31.525470 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.525460    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 20 15:31:31.525470 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.525470    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 20 15:31:31.527382 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:31.527358    2568 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 20 15:31:31.527504 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:31.527463    2568 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-117.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 20 15:31:31.529085 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.529073    2568 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 20 15:31:31.529135 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.529116    2568 server.go:1295] "Started kubelet"
May 20 15:31:31.529237 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.529188    2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 20 15:31:31.529284 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.529229    2568 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 20 15:31:31.529332 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.529294    2568 server_v1.go:47] "podresources" method="list" useActivePods=true
May 20 15:31:31.529959 ip-10-0-129-117 systemd[1]: Started Kubernetes Kubelet.
May 20 15:31:31.530718 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.530704    2568 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 20 15:31:31.532812 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.532794    2568 server.go:317] "Adding debug handlers to kubelet server"
May 20 15:31:31.534608 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.534588    2568 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-117.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 20 15:31:31.535606 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:31.534667    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-117.ec2.internal.18b14fd2169bb446  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-117.ec2.internal,UID:ip-10-0-129-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-117.ec2.internal,},FirstTimestamp:2026-05-20 15:31:31.529086022 +0000 UTC m=+0.467927569,LastTimestamp:2026-05-20 15:31:31.529086022 +0000 UTC m=+0.467927569,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-117.ec2.internal,}"
May 20 15:31:31.536870 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.536852    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 20 15:31:31.537423 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.537401    2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 20 15:31:31.537802 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.537779    2568 volume_manager.go:295] "The desired_state_of_world populator starts"
May 20 15:31:31.537802 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.537803    2568 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 20 15:31:31.537899 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.537858    2568 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 20 15:31:31.537960 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.537945    2568 reconstruct.go:97] "Volume reconstruction finished"
May 20 15:31:31.537960 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.537957    2568 reconciler.go:26] "Reconciler: start to sync state"
May 20 15:31:31.538632 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:31.538611    2568 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 15:31:31.538907 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.538891    2568 factory.go:153] Registering CRI-O factory
May 20 15:31:31.538990 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.538908    2568 factory.go:223] Registration of the crio container factory successfully
May 20 15:31:31.538990 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.538977    2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 20 15:31:31.538990 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.538986    2568 factory.go:55] Registering systemd factory
May 20 15:31:31.539128 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.538994    2568 factory.go:223] Registration of the systemd container factory successfully
May 20 15:31:31.539128 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.539009    2568 factory.go:103] Registering Raw factory
May 20 15:31:31.539128 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.539048    2568 manager.go:1196] Started watching for new ooms in manager
May 20 15:31:31.540067 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.540052    2568 manager.go:319] Starting recovery of all containers
May 20 15:31:31.541531 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:31.541501    2568 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 20 15:31:31.549288 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:31.549140    2568 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 20 15:31:31.549414 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:31.549162    2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-117.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 20 15:31:31.552671 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.552653    2568 manager.go:324] Recovery completed
May 20 15:31:31.556484 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.556467    2568 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:31:31.558426 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.558408    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:31:31.558482 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.558441    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:31:31.558482 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.558452    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientPID"
May 20 15:31:31.558941 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.558926    2568 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 20 15:31:31.558988 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.558942    2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 20 15:31:31.558988 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.558960    2568 state_mem.go:36] "Initialized new in-memory state store"
May 20 15:31:31.560879 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:31.560813    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-117.ec2.internal.18b14fd2185b6a57  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-117.ec2.internal,UID:ip-10-0-129-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-117.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-117.ec2.internal,},FirstTimestamp:2026-05-20 15:31:31.558427223 +0000 UTC m=+0.497268771,LastTimestamp:2026-05-20 15:31:31.558427223 +0000 UTC m=+0.497268771,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-117.ec2.internal,}"
May 20 15:31:31.561384 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.561372    2568 policy_none.go:49] "None policy: Start"
May 20 15:31:31.561419 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.561392    2568 memory_manager.go:186] "Starting memorymanager" policy="None"
May 20 15:31:31.561419 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.561402    2568 state_mem.go:35] "Initializing new in-memory state store"
May 20 15:31:31.567949 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:31.567878    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-117.ec2.internal.18b14fd2185bb445  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-117.ec2.internal,UID:ip-10-0-129-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-117.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-117.ec2.internal,},FirstTimestamp:2026-05-20 15:31:31.558446149 +0000 UTC m=+0.497287697,LastTimestamp:2026-05-20 15:31:31.558446149 +0000 UTC m=+0.497287697,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-117.ec2.internal,}"
May 20 15:31:31.575223 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.575204    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lqsd8"
May 20 15:31:31.580124 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:31.580061    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-117.ec2.internal.18b14fd2185bd948  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-117.ec2.internal,UID:ip-10-0-129-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-117.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-117.ec2.internal,},FirstTimestamp:2026-05-20 15:31:31.558455624 +0000 UTC m=+0.497297173,LastTimestamp:2026-05-20 15:31:31.558455624 +0000 UTC m=+0.497297173,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-117.ec2.internal,}"
May 20 15:31:31.584186 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.584163    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lqsd8"
May 20 15:31:31.608661 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.608644    2568 manager.go:341] "Starting Device Plugin manager"
May 20 15:31:31.614122 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:31.608685    2568 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 20 15:31:31.614122 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.608696    2568 server.go:85] "Starting device plugin registration server"
May 20 15:31:31.614122 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.609091    2568 eviction_manager.go:189] "Eviction manager: starting control loop"
May 20 15:31:31.614122 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.609102    2568 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 20 15:31:31.614122 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.609218    2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 20 15:31:31.614122 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.609299    2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 20 15:31:31.614122 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.609310    2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 20 15:31:31.614122 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:31.609787    2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 20 15:31:31.614122 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:31.609834    2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 15:31:31.637140 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.637110    2568 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 20 15:31:31.638533 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.638512    2568 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 20 15:31:31.638643 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.638539    2568 status_manager.go:230] "Starting to sync pod status with apiserver"
May 20 15:31:31.638643 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.638561    2568 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 20 15:31:31.638643 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.638570    2568 kubelet.go:2451] "Starting kubelet main sync loop"
May 20 15:31:31.638643 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:31.638617    2568 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 20 15:31:31.641389 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.641365    2568 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:31.709688 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.709573    2568 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:31:31.712878 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.712859    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:31:31.713007 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.712895    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:31:31.713007 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.712910    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientPID"
May 20 15:31:31.713007 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.712950    2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-117.ec2.internal"
May 20 15:31:31.722122 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.722104    2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-117.ec2.internal"
May 20 15:31:31.722237 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:31.722128    2568 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-117.ec2.internal\": node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 15:31:31.737380 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:31.737360    2568 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 15:31:31.739343 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.739310    2568 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal"]
May 20 15:31:31.739422 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.739384    2568 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:31:31.742311 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.742294    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:31:31.742392 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.742331    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:31:31.742392 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.742347    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientPID"
May 20 15:31:31.744093 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.744078    2568 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:31:31.744236 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.744220    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
May 20 15:31:31.744288 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.744257    2568 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:31:31.744855 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.744839    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:31:31.744902 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.744862    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:31:31.744902 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.744871    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientPID"
May 20 15:31:31.744962 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.744899    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:31:31.744962 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.744921    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:31:31.744962 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.744930    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientPID"
May 20 15:31:31.746648 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.746628    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal"
May 20 15:31:31.746737 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.746654    2568 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:31:31.747397 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.747383    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:31:31.747490 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.747411    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:31:31.747490 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.747424    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientPID"
May 20 15:31:31.770610 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:31.770587    2568 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-117.ec2.internal\" not found" node="ip-10-0-129-117.ec2.internal"
May 20 15:31:31.775148 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:31.775131    2568 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-117.ec2.internal\" not found" node="ip-10-0-129-117.ec2.internal"
May 20 15:31:31.837865 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:31.837825    2568 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 15:31:31.938312 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:31.938284    2568 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 15:31:31.939464 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.939438    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a720ccbb8d80c2c23e6edde668812484-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal\" (UID: \"a720ccbb8d80c2c23e6edde668812484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
May 20 15:31:31.939561 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.939487    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a720ccbb8d80c2c23e6edde668812484-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal\" (UID: \"a720ccbb8d80c2c23e6edde668812484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
May 20 15:31:31.939561 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:31.939529    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f81a807c436a78d0ae12cc5e66b2e05-config\") pod \"kube-apiserver-proxy-ip-10-0-129-117.ec2.internal\" (UID: \"6f81a807c436a78d0ae12cc5e66b2e05\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal"
May 20 15:31:32.038934 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:32.038850    2568 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 15:31:32.040055 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.040036    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f81a807c436a78d0ae12cc5e66b2e05-config\") pod \"kube-apiserver-proxy-ip-10-0-129-117.ec2.internal\" (UID: \"6f81a807c436a78d0ae12cc5e66b2e05\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal"
May 20 15:31:32.040104 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.040065    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a720ccbb8d80c2c23e6edde668812484-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal\" (UID: \"a720ccbb8d80c2c23e6edde668812484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
May 20 15:31:32.040181 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.040088    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a720ccbb8d80c2c23e6edde668812484-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal\" (UID: \"a720ccbb8d80c2c23e6edde668812484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
May 20 15:31:32.040181 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.040114    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f81a807c436a78d0ae12cc5e66b2e05-config\") pod \"kube-apiserver-proxy-ip-10-0-129-117.ec2.internal\" (UID: \"6f81a807c436a78d0ae12cc5e66b2e05\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal"
May 20 15:31:32.040181 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.040120    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a720ccbb8d80c2c23e6edde668812484-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal\" (UID: \"a720ccbb8d80c2c23e6edde668812484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
May 20 15:31:32.040181 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.040133    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a720ccbb8d80c2c23e6edde668812484-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal\" (UID: \"a720ccbb8d80c2c23e6edde668812484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
May 20 15:31:32.073171 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.073140    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
May 20 15:31:32.077940 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.077919    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal"
May 20 15:31:32.139979 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:32.139944    2568 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 15:31:32.240378 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:32.240346    2568 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 15:31:32.340760 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:32.340731    2568 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 15:31:32.441154 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:32.441123    2568 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 15:31:32.447356 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.447334    2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 20 15:31:32.447522 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.447504    2568 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 20 15:31:32.528642 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.528607    2568 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:32.537095 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.537075    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 20 15:31:32.541458 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:32.541432    2568 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 15:31:32.545776 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.545756    2568 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 20 15:31:32.561817 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:32.561783    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f81a807c436a78d0ae12cc5e66b2e05.slice/crio-afaaceeca51543319fc9d29cbdec3eec5375546882080e519b4bc79ca301f0ce WatchSource:0}: Error finding container afaaceeca51543319fc9d29cbdec3eec5375546882080e519b4bc79ca301f0ce: Status 404 returned error can't find the container with id afaaceeca51543319fc9d29cbdec3eec5375546882080e519b4bc79ca301f0ce
May 20 15:31:32.562117 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:32.562090    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda720ccbb8d80c2c23e6edde668812484.slice/crio-9f8f5be1b468002c58171bf9cc5876ea338a0c9ac89ab5cb4c80e30905a2345a WatchSource:0}: Error finding container 9f8f5be1b468002c58171bf9cc5876ea338a0c9ac89ab5cb4c80e30905a2345a: Status 404 returned error can't find the container with id 9f8f5be1b468002c58171bf9cc5876ea338a0c9ac89ab5cb4c80e30905a2345a
May 20 15:31:32.567573 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.567552    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:31:32.586190 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.586158    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-19 15:26:31 +0000 UTC" deadline="2028-02-12 12:14:27.976678946 +0000 UTC"
May 20 15:31:32.586190 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.586187    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15188h42m55.390494816s"
May 20 15:31:32.597185 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.597137    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xp6pt"
May 20 15:31:32.604235 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.604208    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xp6pt"
May 20 15:31:32.641511 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:32.641487    2568 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 15:31:32.642402 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.642345    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal" event={"ID":"6f81a807c436a78d0ae12cc5e66b2e05","Type":"ContainerStarted","Data":"afaaceeca51543319fc9d29cbdec3eec5375546882080e519b4bc79ca301f0ce"}
May 20 15:31:32.643227 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.643207    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal" event={"ID":"a720ccbb8d80c2c23e6edde668812484","Type":"ContainerStarted","Data":"9f8f5be1b468002c58171bf9cc5876ea338a0c9ac89ab5cb4c80e30905a2345a"}
May 20 15:31:32.682293 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.682270    2568 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:32.742434 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:32.742399    2568 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 15:31:32.799760 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.799733    2568 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:32.838619 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.838590    2568 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
May 20 15:31:32.851562 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.851493    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 20 15:31:32.852419 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.852402    2568 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal"
May 20 15:31:32.859295 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:32.859275    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 20 15:31:33.515772 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.515739    2568 apiserver.go:52] "Watching apiserver"
May 20 15:31:33.524487 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.524465    2568 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 20 15:31:33.526504 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.526477    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-gqhp4","kube-system/konnectivity-agent-g8s5k","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n","openshift-image-registry/node-ca-l24nh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal","openshift-multus/multus-additional-cni-plugins-kvrb6","kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal","openshift-cluster-node-tuning-operator/tuned-th4j4","openshift-multus/multus-bjx28","openshift-multus/network-metrics-daemon-skk8q","openshift-network-diagnostics/network-check-target-r9x8t","openshift-network-operator/iptables-alerter-kffdt"]
May 20 15:31:33.528491 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.528469    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.529651 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.529632    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g8s5k"
May 20 15:31:33.530923 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.530900    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 20 15:31:33.530923 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.530912    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.531728 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.531311    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 20 15:31:33.531728 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.531342    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 20 15:31:33.531728 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.531449    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 20 15:31:33.531728 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.531566    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c69dp\""
May 20 15:31:33.531728 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.531598    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 20 15:31:33.532080 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.532060    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l24nh"
May 20 15:31:33.532256 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.532235    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 20 15:31:33.532520 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.532500    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 20 15:31:33.532520 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.532520    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wsl4c\""
May 20 15:31:33.533105 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.532861    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 20 15:31:33.533105 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.533032    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 20 15:31:33.533351 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.533330    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 20 15:31:33.533491 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.533366    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-llc6r\""
May 20 15:31:33.533964 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.533887    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 20 15:31:33.534118 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.534096    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 20 15:31:33.534276 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.534258    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-45dhx\""
May 20 15:31:33.534321 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.534289    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 20 15:31:33.534354 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.534326    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 20 15:31:33.534754 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.534737    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.534842 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.534824    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.537033 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.536978    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 20 15:31:33.538007 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.537540    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 20 15:31:33.538007 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.537571    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 20 15:31:33.538007 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.537605    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 20 15:31:33.538007 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.537613    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 20 15:31:33.538007 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.537624    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 20 15:31:33.538007 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.537681    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lbm42\""
May 20 15:31:33.538332 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.538055    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 20 15:31:33.538809 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.538785    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bjx28"
May 20 15:31:33.539195 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.539176    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j9lkd\""
May 20 15:31:33.540611 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.540110    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:33.540611 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:33.540177    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skk8q" podUID="3022a27a-6112-46ee-a54b-c5af5bee2a1f"
May 20 15:31:33.540737 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.540666    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 20 15:31:33.540825 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.540807    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-45hb8\""
May 20 15:31:33.541650 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.541631    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:31:33.541746 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:33.541707    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9x8t" podUID="69c7e426-cc15-4efc-afbe-77377c1c3779"
May 20 15:31:33.542903 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.542870    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kffdt"
May 20 15:31:33.545140 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.545117    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 20 15:31:33.545224 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.545154    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 20 15:31:33.545224 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.545214    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 20 15:31:33.545317 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.545118    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jj6zl\""
May 20 15:31:33.546435 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546417    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv4vx\" (UniqueName: \"kubernetes.io/projected/2fe4adfc-e1dd-4a70-a161-f0d525f3ca3b-kube-api-access-wv4vx\") pod \"node-ca-l24nh\" (UID: \"2fe4adfc-e1dd-4a70-a161-f0d525f3ca3b\") " pod="openshift-image-registry/node-ca-l24nh"
May 20 15:31:33.546542 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546442    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b49cf359-1bf7-485e-8ed9-9171a4111b2c-sys-fs\") pod \"aws-ebs-csi-driver-node-pzb4n\" (UID: \"b49cf359-1bf7-485e-8ed9-9171a4111b2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.546542 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546457    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-host-slash\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.546542 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546472    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-run\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.546542 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546490    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-hostroot\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.546542 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546511    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-multus-daemon-config\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.546804 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546535    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/799a84c1-8372-4b99-a34d-105048106d5d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.546804 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546567    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-host-run-netns\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.546804 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546589    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-run-openvswitch\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.546804 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546609    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6bf94e80-c129-435d-a864-29e97ca3634f-ovn-node-metrics-cert\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.546804 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546627    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-etc-kubernetes\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.546804 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546649    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-var-lib-kubelet\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.546804 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546695    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ccfe95d-efc9-46ec-aaac-da92e41a5574-etc-tuned\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.546804 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546738    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ccfe95d-efc9-46ec-aaac-da92e41a5574-tmp\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.546804 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546770    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b49cf359-1bf7-485e-8ed9-9171a4111b2c-socket-dir\") pod \"aws-ebs-csi-driver-node-pzb4n\" (UID: \"b49cf359-1bf7-485e-8ed9-9171a4111b2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.546804 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546795    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtpz5\" (UniqueName: \"kubernetes.io/projected/6bf94e80-c129-435d-a864-29e97ca3634f-kube-api-access-dtpz5\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.547395 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546819    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-host-cni-bin\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.547395 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546843    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-etc-systemd\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.547395 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546865    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2fe4adfc-e1dd-4a70-a161-f0d525f3ca3b-serviceca\") pod \"node-ca-l24nh\" (UID: \"2fe4adfc-e1dd-4a70-a161-f0d525f3ca3b\") " pod="openshift-image-registry/node-ca-l24nh"
May 20 15:31:33.547395 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546885    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-host\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.547395 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546907    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf5k6\" (UniqueName: \"kubernetes.io/projected/3ccfe95d-efc9-46ec-aaac-da92e41a5574-kube-api-access-sf5k6\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.547395 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546929    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6bf94e80-c129-435d-a864-29e97ca3634f-ovnkube-config\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.547395 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.546959    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-etc-modprobe-d\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.547395 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547038    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-host-run-netns\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.547395 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547064    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-host-var-lib-cni-bin\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.547395 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547088    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs7d6\" (UniqueName: \"kubernetes.io/projected/b49cf359-1bf7-485e-8ed9-9171a4111b2c-kube-api-access-vs7d6\") pod \"aws-ebs-csi-driver-node-pzb4n\" (UID: \"b49cf359-1bf7-485e-8ed9-9171a4111b2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.547395 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547111    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-multus-conf-dir\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.547395 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547129    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f5adfb93-88e3-4b63-839c-31c7a0793b04-agent-certs\") pod \"konnectivity-agent-g8s5k\" (UID: \"f5adfb93-88e3-4b63-839c-31c7a0793b04\") " pod="kube-system/konnectivity-agent-g8s5k"
May 20 15:31:33.547395 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547168    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-host-var-lib-kubelet\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.547395 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547191    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fe4adfc-e1dd-4a70-a161-f0d525f3ca3b-host\") pod \"node-ca-l24nh\" (UID: \"2fe4adfc-e1dd-4a70-a161-f0d525f3ca3b\") " pod="openshift-image-registry/node-ca-l24nh"
May 20 15:31:33.547395 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547208    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-etc-sysconfig\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.547395 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547239    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-multus-socket-dir-parent\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.547395 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547256    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk84b\" (UniqueName: \"kubernetes.io/projected/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-kube-api-access-kk84b\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.547904 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547277    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/799a84c1-8372-4b99-a34d-105048106d5d-cnibin\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.547904 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547291    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b49cf359-1bf7-485e-8ed9-9171a4111b2c-registration-dir\") pod \"aws-ebs-csi-driver-node-pzb4n\" (UID: \"b49cf359-1bf7-485e-8ed9-9171a4111b2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.547904 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547306    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6bf94e80-c129-435d-a864-29e97ca3634f-ovnkube-script-lib\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.547904 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547321    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/799a84c1-8372-4b99-a34d-105048106d5d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.547904 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547355    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/799a84c1-8372-4b99-a34d-105048106d5d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.547904 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547388    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-run-systemd\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.547904 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547411    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-var-lib-openvswitch\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.547904 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547431    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-host-run-ovn-kubernetes\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.547904 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547445    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b49cf359-1bf7-485e-8ed9-9171a4111b2c-device-dir\") pod \"aws-ebs-csi-driver-node-pzb4n\" (UID: \"b49cf359-1bf7-485e-8ed9-9171a4111b2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.547904 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547462    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b49cf359-1bf7-485e-8ed9-9171a4111b2c-etc-selinux\") pod \"aws-ebs-csi-driver-node-pzb4n\" (UID: \"b49cf359-1bf7-485e-8ed9-9171a4111b2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.547904 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547493    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/799a84c1-8372-4b99-a34d-105048106d5d-os-release\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.547904 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547513    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/799a84c1-8372-4b99-a34d-105048106d5d-cni-binary-copy\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.547904 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547530    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f5adfb93-88e3-4b63-839c-31c7a0793b04-konnectivity-ca\") pod \"konnectivity-agent-g8s5k\" (UID: \"f5adfb93-88e3-4b63-839c-31c7a0793b04\") " pod="kube-system/konnectivity-agent-g8s5k"
May 20 15:31:33.547904 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547552    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-run-ovn\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.547904 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547574    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-system-cni-dir\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.547904 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547592    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/799a84c1-8372-4b99-a34d-105048106d5d-system-cni-dir\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.548568 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547609    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lfq8\" (UniqueName: \"kubernetes.io/projected/799a84c1-8372-4b99-a34d-105048106d5d-kube-api-access-9lfq8\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.548568 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547635    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-etc-openvswitch\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.548568 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547655    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-etc-sysctl-d\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.548568 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547698    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-lib-modules\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.548568 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547737    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-cni-binary-copy\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.548568 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547763    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-host-run-k8s-cni-cncf-io\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.548568 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547787    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-host-var-lib-cni-multus\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.548568 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547803    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-host-run-multus-certs\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.548568 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547845    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-etc-kubernetes\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.548568 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547871    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-node-log\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.548568 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547886    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-sys\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.548568 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547902    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-log-socket\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.548568 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547916    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6bf94e80-c129-435d-a864-29e97ca3634f-env-overrides\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.548568 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547932    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-etc-sysctl-conf\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.548568 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547945    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-multus-cni-dir\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.548568 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547970    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-cnibin\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.548568 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.547999    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b49cf359-1bf7-485e-8ed9-9171a4111b2c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pzb4n\" (UID: \"b49cf359-1bf7-485e-8ed9-9171a4111b2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.549435 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.548034    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-systemd-units\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.549435 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.548059    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-host-cni-netd\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.549435 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.548087    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.549435 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.548112    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-os-release\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.549435 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.548139    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-host-kubelet\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.603055 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.603015    2568 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:33.605784 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.605752    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 15:26:32 +0000 UTC" deadline="2027-11-10 11:22:03.218663338 +0000 UTC"
May 20 15:31:33.605784 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.605777    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12931h50m29.612889084s"
May 20 15:31:33.638920 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.638886    2568 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 20 15:31:33.648533 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.648506    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6bf94e80-c129-435d-a864-29e97ca3634f-ovnkube-script-lib\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.648533 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.648538    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/799a84c1-8372-4b99-a34d-105048106d5d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.648749 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.648557    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/799a84c1-8372-4b99-a34d-105048106d5d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.648749 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.648586    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5tg9\" (UniqueName: \"kubernetes.io/projected/6fefe56a-44d8-47d6-bae5-3926f0bc3b1c-kube-api-access-k5tg9\") pod \"iptables-alerter-kffdt\" (UID: \"6fefe56a-44d8-47d6-bae5-3926f0bc3b1c\") " pod="openshift-network-operator/iptables-alerter-kffdt"
May 20 15:31:33.648749 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.648668    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-run-systemd\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.648749 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.648710    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-var-lib-openvswitch\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.648749 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.648737    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-host-run-ovn-kubernetes\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.648956 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.648749    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/799a84c1-8372-4b99-a34d-105048106d5d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.648956 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.648762    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b49cf359-1bf7-485e-8ed9-9171a4111b2c-device-dir\") pod \"aws-ebs-csi-driver-node-pzb4n\" (UID: \"b49cf359-1bf7-485e-8ed9-9171a4111b2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.648956 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.648769    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-run-systemd\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.648956 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.648787    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b49cf359-1bf7-485e-8ed9-9171a4111b2c-etc-selinux\") pod \"aws-ebs-csi-driver-node-pzb4n\" (UID: \"b49cf359-1bf7-485e-8ed9-9171a4111b2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.648956 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.648804    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-var-lib-openvswitch\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.648956 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.648812    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/799a84c1-8372-4b99-a34d-105048106d5d-os-release\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.648956 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.648832    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-host-run-ovn-kubernetes\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.648956 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.648855    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b49cf359-1bf7-485e-8ed9-9171a4111b2c-device-dir\") pod \"aws-ebs-csi-driver-node-pzb4n\" (UID: \"b49cf359-1bf7-485e-8ed9-9171a4111b2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.648956 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.648885    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/799a84c1-8372-4b99-a34d-105048106d5d-cni-binary-copy\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.648956 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.648891    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b49cf359-1bf7-485e-8ed9-9171a4111b2c-etc-selinux\") pod \"aws-ebs-csi-driver-node-pzb4n\" (UID: \"b49cf359-1bf7-485e-8ed9-9171a4111b2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.648956 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.648893    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/799a84c1-8372-4b99-a34d-105048106d5d-os-release\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.648956 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.648910    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f5adfb93-88e3-4b63-839c-31c7a0793b04-konnectivity-ca\") pod \"konnectivity-agent-g8s5k\" (UID: \"f5adfb93-88e3-4b63-839c-31c7a0793b04\") " pod="kube-system/konnectivity-agent-g8s5k"
May 20 15:31:33.648956 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.648937    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-run-ovn\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.648956 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.648961    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-system-cni-dir\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.649615 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.648998    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/799a84c1-8372-4b99-a34d-105048106d5d-system-cni-dir\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.649615 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649044    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9lfq8\" (UniqueName: \"kubernetes.io/projected/799a84c1-8372-4b99-a34d-105048106d5d-kube-api-access-9lfq8\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.649615 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649070    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-etc-openvswitch\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.649615 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649093    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-etc-sysctl-d\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.649615 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649115    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-lib-modules\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.649615 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649136    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-cni-binary-copy\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.649615 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649158    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-host-run-k8s-cni-cncf-io\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.649615 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649164    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-system-cni-dir\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.649615 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649183    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-host-var-lib-cni-multus\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.649615 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649209    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-host-run-multus-certs\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.649615 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649223    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-run-ovn\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.649615 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649230    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-etc-kubernetes\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.649615 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649253    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-node-log\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.649615 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649266    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6bf94e80-c129-435d-a864-29e97ca3634f-ovnkube-script-lib\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.649615 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649276    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-sys\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.649615 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649301    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-log-socket\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.649615 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649303    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/799a84c1-8372-4b99-a34d-105048106d5d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.650385 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649326    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6bf94e80-c129-435d-a864-29e97ca3634f-env-overrides\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.650385 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649299    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-etc-openvswitch\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.650385 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649353    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-etc-sysctl-conf\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.650385 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649369    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-host-run-multus-certs\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.650385 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649379    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-multus-cni-dir\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.650385 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649383    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-host-run-k8s-cni-cncf-io\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.650385 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649403    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-lib-modules\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.650385 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649408    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-host-var-lib-cni-multus\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.650385 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649415    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-cnibin\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.650385 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649422    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/799a84c1-8372-4b99-a34d-105048106d5d-cni-binary-copy\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.650385 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649441    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b49cf359-1bf7-485e-8ed9-9171a4111b2c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pzb4n\" (UID: \"b49cf359-1bf7-485e-8ed9-9171a4111b2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.650385 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649456    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-log-socket\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.650385 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649470    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r57xm\" (UniqueName: \"kubernetes.io/projected/3022a27a-6112-46ee-a54b-c5af5bee2a1f-kube-api-access-r57xm\") pod \"network-metrics-daemon-skk8q\" (UID: \"3022a27a-6112-46ee-a54b-c5af5bee2a1f\") " pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:33.650385 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649483    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-multus-cni-dir\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.650385 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649488    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-etc-kubernetes\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.650385 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649502    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-systemd-units\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.650385 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649512    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f5adfb93-88e3-4b63-839c-31c7a0793b04-konnectivity-ca\") pod \"konnectivity-agent-g8s5k\" (UID: \"f5adfb93-88e3-4b63-839c-31c7a0793b04\") " pod="kube-system/konnectivity-agent-g8s5k"
May 20 15:31:33.650385 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649529    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-host-cni-netd\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.651076 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649533    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-etc-sysctl-d\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.651076 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649523    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-node-log\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.651076 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649560    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.651076 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649573    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-sys\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.651076 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649588    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-os-release\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.651076 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649597    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b49cf359-1bf7-485e-8ed9-9171a4111b2c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pzb4n\" (UID: \"b49cf359-1bf7-485e-8ed9-9171a4111b2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.651076 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649595    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-cnibin\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.651076 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649603    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-etc-sysctl-conf\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.651076 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649615    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-host-kubelet\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.651076 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649615    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-host-cni-netd\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.651076 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649653    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-host-kubelet\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.651076 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649665    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wv4vx\" (UniqueName: \"kubernetes.io/projected/2fe4adfc-e1dd-4a70-a161-f0d525f3ca3b-kube-api-access-wv4vx\") pod \"node-ca-l24nh\" (UID: \"2fe4adfc-e1dd-4a70-a161-f0d525f3ca3b\") " pod="openshift-image-registry/node-ca-l24nh"
May 20 15:31:33.651076 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649651    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.651076 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649668    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-os-release\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.651076 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649689    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-systemd-units\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.651076 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649692    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b49cf359-1bf7-485e-8ed9-9171a4111b2c-sys-fs\") pod \"aws-ebs-csi-driver-node-pzb4n\" (UID: \"b49cf359-1bf7-485e-8ed9-9171a4111b2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.651076 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649739    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-host-slash\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.651076 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649742    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b49cf359-1bf7-485e-8ed9-9171a4111b2c-sys-fs\") pod \"aws-ebs-csi-driver-node-pzb4n\" (UID: \"b49cf359-1bf7-485e-8ed9-9171a4111b2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.651780 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649764    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-run\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.651780 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649765    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6bf94e80-c129-435d-a864-29e97ca3634f-env-overrides\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.651780 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649785    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-host-slash\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.651780 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649788    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-hostroot\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.651780 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649807    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-run\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.651780 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649818    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-multus-daemon-config\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.651780 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649822    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-hostroot\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.651780 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649862    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-cni-binary-copy\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.651780 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649904    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/799a84c1-8372-4b99-a34d-105048106d5d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.651780 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649935    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-host-run-netns\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.651780 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649959    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-run-openvswitch\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.651780 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649932    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/799a84c1-8372-4b99-a34d-105048106d5d-system-cni-dir\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.651780 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649983    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6bf94e80-c129-435d-a864-29e97ca3634f-ovn-node-metrics-cert\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.651780 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.649992    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-host-run-netns\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.651780 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650008    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-etc-kubernetes\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.651780 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650033    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-run-openvswitch\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.651780 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650048    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-var-lib-kubelet\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.651780 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650073    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ccfe95d-efc9-46ec-aaac-da92e41a5574-etc-tuned\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.652734 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650083    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-etc-kubernetes\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.652734 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650095    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ccfe95d-efc9-46ec-aaac-da92e41a5574-tmp\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.652734 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650126    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-var-lib-kubelet\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.652734 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650152    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b49cf359-1bf7-485e-8ed9-9171a4111b2c-socket-dir\") pod \"aws-ebs-csi-driver-node-pzb4n\" (UID: \"b49cf359-1bf7-485e-8ed9-9171a4111b2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.652734 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650179    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dtpz5\" (UniqueName: \"kubernetes.io/projected/6bf94e80-c129-435d-a864-29e97ca3634f-kube-api-access-dtpz5\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.652734 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650203    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-host-cni-bin\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.652734 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650227    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-etc-systemd\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.652734 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650250    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2fe4adfc-e1dd-4a70-a161-f0d525f3ca3b-serviceca\") pod \"node-ca-l24nh\" (UID: \"2fe4adfc-e1dd-4a70-a161-f0d525f3ca3b\") " pod="openshift-image-registry/node-ca-l24nh"
May 20 15:31:33.652734 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650272    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-host\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.652734 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650297    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sf5k6\" (UniqueName: \"kubernetes.io/projected/3ccfe95d-efc9-46ec-aaac-da92e41a5574-kube-api-access-sf5k6\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.652734 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650318    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slmsd\" (UniqueName: \"kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd\") pod \"network-check-target-r9x8t\" (UID: \"69c7e426-cc15-4efc-afbe-77377c1c3779\") " pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:31:33.652734 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650327    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6bf94e80-c129-435d-a864-29e97ca3634f-host-cni-bin\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.652734 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650336    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6bf94e80-c129-435d-a864-29e97ca3634f-ovnkube-config\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.652734 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650357    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-etc-modprobe-d\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.652734 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650372    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-host-run-netns\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.652734 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650389    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-host-var-lib-cni-bin\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.652734 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650404    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vs7d6\" (UniqueName: \"kubernetes.io/projected/b49cf359-1bf7-485e-8ed9-9171a4111b2c-kube-api-access-vs7d6\") pod \"aws-ebs-csi-driver-node-pzb4n\" (UID: \"b49cf359-1bf7-485e-8ed9-9171a4111b2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.653503 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650421    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6fefe56a-44d8-47d6-bae5-3926f0bc3b1c-iptables-alerter-script\") pod \"iptables-alerter-kffdt\" (UID: \"6fefe56a-44d8-47d6-bae5-3926f0bc3b1c\") " pod="openshift-network-operator/iptables-alerter-kffdt"
May 20 15:31:33.653503 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650436    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fefe56a-44d8-47d6-bae5-3926f0bc3b1c-host-slash\") pod \"iptables-alerter-kffdt\" (UID: \"6fefe56a-44d8-47d6-bae5-3926f0bc3b1c\") " pod="openshift-network-operator/iptables-alerter-kffdt"
May 20 15:31:33.653503 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650433    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/799a84c1-8372-4b99-a34d-105048106d5d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.653503 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650319    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-multus-daemon-config\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.653503 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650484    2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 20 15:31:33.653503 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650503    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-host\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.653503 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650516    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-etc-modprobe-d\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.653503 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650540    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-multus-conf-dir\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.653503 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650551    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-host-var-lib-cni-bin\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.653503 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650592    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-etc-systemd\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.653503 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650444    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b49cf359-1bf7-485e-8ed9-9171a4111b2c-socket-dir\") pod \"aws-ebs-csi-driver-node-pzb4n\" (UID: \"b49cf359-1bf7-485e-8ed9-9171a4111b2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.653503 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650450    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-multus-conf-dir\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.653503 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650715    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f5adfb93-88e3-4b63-839c-31c7a0793b04-agent-certs\") pod \"konnectivity-agent-g8s5k\" (UID: \"f5adfb93-88e3-4b63-839c-31c7a0793b04\") " pod="kube-system/konnectivity-agent-g8s5k"
May 20 15:31:33.653503 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650732    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-host-var-lib-kubelet\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.653503 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650770    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-host-var-lib-kubelet\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.653503 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650801    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs\") pod \"network-metrics-daemon-skk8q\" (UID: \"3022a27a-6112-46ee-a54b-c5af5bee2a1f\") " pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:33.653503 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650826    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fe4adfc-e1dd-4a70-a161-f0d525f3ca3b-host\") pod \"node-ca-l24nh\" (UID: \"2fe4adfc-e1dd-4a70-a161-f0d525f3ca3b\") " pod="openshift-image-registry/node-ca-l24nh"
May 20 15:31:33.653503 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650854    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-etc-sysconfig\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.656228 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650906    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-multus-socket-dir-parent\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.656228 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650927    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kk84b\" (UniqueName: \"kubernetes.io/projected/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-kube-api-access-kk84b\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.656228 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650947    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/799a84c1-8372-4b99-a34d-105048106d5d-cnibin\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.656228 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650970    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b49cf359-1bf7-485e-8ed9-9171a4111b2c-registration-dir\") pod \"aws-ebs-csi-driver-node-pzb4n\" (UID: \"b49cf359-1bf7-485e-8ed9-9171a4111b2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.656228 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.650974    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6bf94e80-c129-435d-a864-29e97ca3634f-ovnkube-config\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.656228 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.651000    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2fe4adfc-e1dd-4a70-a161-f0d525f3ca3b-serviceca\") pod \"node-ca-l24nh\" (UID: \"2fe4adfc-e1dd-4a70-a161-f0d525f3ca3b\") " pod="openshift-image-registry/node-ca-l24nh"
May 20 15:31:33.656228 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.651045    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-host-run-netns\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.656228 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.651057    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b49cf359-1bf7-485e-8ed9-9171a4111b2c-registration-dir\") pod \"aws-ebs-csi-driver-node-pzb4n\" (UID: \"b49cf359-1bf7-485e-8ed9-9171a4111b2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.656228 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.651079    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ccfe95d-efc9-46ec-aaac-da92e41a5574-etc-sysconfig\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.656228 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.651096    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fe4adfc-e1dd-4a70-a161-f0d525f3ca3b-host\") pod \"node-ca-l24nh\" (UID: \"2fe4adfc-e1dd-4a70-a161-f0d525f3ca3b\") " pod="openshift-image-registry/node-ca-l24nh"
May 20 15:31:33.656228 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.651099    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-multus-socket-dir-parent\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.656228 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.651135    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/799a84c1-8372-4b99-a34d-105048106d5d-cnibin\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.656228 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.654542    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ccfe95d-efc9-46ec-aaac-da92e41a5574-tmp\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.656228 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.654577    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ccfe95d-efc9-46ec-aaac-da92e41a5574-etc-tuned\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.656228 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.654712    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6bf94e80-c129-435d-a864-29e97ca3634f-ovn-node-metrics-cert\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.656228 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.655270    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f5adfb93-88e3-4b63-839c-31c7a0793b04-agent-certs\") pod \"konnectivity-agent-g8s5k\" (UID: \"f5adfb93-88e3-4b63-839c-31c7a0793b04\") " pod="kube-system/konnectivity-agent-g8s5k"
May 20 15:31:33.659962 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.659938    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv4vx\" (UniqueName: \"kubernetes.io/projected/2fe4adfc-e1dd-4a70-a161-f0d525f3ca3b-kube-api-access-wv4vx\") pod \"node-ca-l24nh\" (UID: \"2fe4adfc-e1dd-4a70-a161-f0d525f3ca3b\") " pod="openshift-image-registry/node-ca-l24nh"
May 20 15:31:33.660409 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.660387    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lfq8\" (UniqueName: \"kubernetes.io/projected/799a84c1-8372-4b99-a34d-105048106d5d-kube-api-access-9lfq8\") pod \"multus-additional-cni-plugins-kvrb6\" (UID: \"799a84c1-8372-4b99-a34d-105048106d5d\") " pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.660823 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.660751    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf5k6\" (UniqueName: \"kubernetes.io/projected/3ccfe95d-efc9-46ec-aaac-da92e41a5574-kube-api-access-sf5k6\") pod \"tuned-th4j4\" (UID: \"3ccfe95d-efc9-46ec-aaac-da92e41a5574\") " pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.660823 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.660774    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk84b\" (UniqueName: \"kubernetes.io/projected/d3f5b3cf-5e4f-49d9-ae81-f81badb12283-kube-api-access-kk84b\") pod \"multus-bjx28\" (UID: \"d3f5b3cf-5e4f-49d9-ae81-f81badb12283\") " pod="openshift-multus/multus-bjx28"
May 20 15:31:33.660984 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.660949    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs7d6\" (UniqueName: \"kubernetes.io/projected/b49cf359-1bf7-485e-8ed9-9171a4111b2c-kube-api-access-vs7d6\") pod \"aws-ebs-csi-driver-node-pzb4n\" (UID: \"b49cf359-1bf7-485e-8ed9-9171a4111b2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.661204 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.661181    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtpz5\" (UniqueName: \"kubernetes.io/projected/6bf94e80-c129-435d-a864-29e97ca3634f-kube-api-access-dtpz5\") pod \"ovnkube-node-gqhp4\" (UID: \"6bf94e80-c129-435d-a864-29e97ca3634f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.751274 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.751238    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-slmsd\" (UniqueName: \"kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd\") pod \"network-check-target-r9x8t\" (UID: \"69c7e426-cc15-4efc-afbe-77377c1c3779\") " pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:31:33.751274 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.751277    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6fefe56a-44d8-47d6-bae5-3926f0bc3b1c-iptables-alerter-script\") pod \"iptables-alerter-kffdt\" (UID: \"6fefe56a-44d8-47d6-bae5-3926f0bc3b1c\") " pod="openshift-network-operator/iptables-alerter-kffdt"
May 20 15:31:33.751475 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.751295    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fefe56a-44d8-47d6-bae5-3926f0bc3b1c-host-slash\") pod \"iptables-alerter-kffdt\" (UID: \"6fefe56a-44d8-47d6-bae5-3926f0bc3b1c\") " pod="openshift-network-operator/iptables-alerter-kffdt"
May 20 15:31:33.751475 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.751350    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fefe56a-44d8-47d6-bae5-3926f0bc3b1c-host-slash\") pod \"iptables-alerter-kffdt\" (UID: \"6fefe56a-44d8-47d6-bae5-3926f0bc3b1c\") " pod="openshift-network-operator/iptables-alerter-kffdt"
May 20 15:31:33.751475 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.751414    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs\") pod \"network-metrics-daemon-skk8q\" (UID: \"3022a27a-6112-46ee-a54b-c5af5bee2a1f\") " pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:33.751475 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.751456    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k5tg9\" (UniqueName: \"kubernetes.io/projected/6fefe56a-44d8-47d6-bae5-3926f0bc3b1c-kube-api-access-k5tg9\") pod \"iptables-alerter-kffdt\" (UID: \"6fefe56a-44d8-47d6-bae5-3926f0bc3b1c\") " pod="openshift-network-operator/iptables-alerter-kffdt"
May 20 15:31:33.751618 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.751491    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r57xm\" (UniqueName: \"kubernetes.io/projected/3022a27a-6112-46ee-a54b-c5af5bee2a1f-kube-api-access-r57xm\") pod \"network-metrics-daemon-skk8q\" (UID: \"3022a27a-6112-46ee-a54b-c5af5bee2a1f\") " pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:33.751618 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:33.751584    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:33.751760 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:33.751694    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs podName:3022a27a-6112-46ee-a54b-c5af5bee2a1f nodeName:}" failed. No retries permitted until 2026-05-20 15:31:34.251639771 +0000 UTC m=+3.190481330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs") pod "network-metrics-daemon-skk8q" (UID: "3022a27a-6112-46ee-a54b-c5af5bee2a1f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:33.751830 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.751802    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6fefe56a-44d8-47d6-bae5-3926f0bc3b1c-iptables-alerter-script\") pod \"iptables-alerter-kffdt\" (UID: \"6fefe56a-44d8-47d6-bae5-3926f0bc3b1c\") " pod="openshift-network-operator/iptables-alerter-kffdt"
May 20 15:31:33.757841 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:33.757816    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:31:33.757841 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:33.757843    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:31:33.757841 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:33.757855    2568 projected.go:194] Error preparing data for projected volume kube-api-access-slmsd for pod openshift-network-diagnostics/network-check-target-r9x8t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:33.758077 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:33.757921    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd podName:69c7e426-cc15-4efc-afbe-77377c1c3779 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:34.257902349 +0000 UTC m=+3.196743909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-slmsd" (UniqueName: "kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd") pod "network-check-target-r9x8t" (UID: "69c7e426-cc15-4efc-afbe-77377c1c3779") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:33.760137 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.760107    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5tg9\" (UniqueName: \"kubernetes.io/projected/6fefe56a-44d8-47d6-bae5-3926f0bc3b1c-kube-api-access-k5tg9\") pod \"iptables-alerter-kffdt\" (UID: \"6fefe56a-44d8-47d6-bae5-3926f0bc3b1c\") " pod="openshift-network-operator/iptables-alerter-kffdt"
May 20 15:31:33.760271 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.760253    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r57xm\" (UniqueName: \"kubernetes.io/projected/3022a27a-6112-46ee-a54b-c5af5bee2a1f-kube-api-access-r57xm\") pod \"network-metrics-daemon-skk8q\" (UID: \"3022a27a-6112-46ee-a54b-c5af5bee2a1f\") " pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:33.843207 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.843162    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:31:33.851013 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.850990    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g8s5k"
May 20 15:31:33.858117 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.858098    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n"
May 20 15:31:33.864742 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.864710    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l24nh"
May 20 15:31:33.871373 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.871354    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kvrb6"
May 20 15:31:33.877965 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.877941    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-th4j4"
May 20 15:31:33.883570 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.883540    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bjx28"
May 20 15:31:33.889073 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:33.889055    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kffdt"
May 20 15:31:34.099531 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:34.099499    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ccfe95d_efc9_46ec_aaac_da92e41a5574.slice/crio-b797a4a79ac0c4bd1cc5741f0aab33ff2ff77338cf0caa43c2c97f7102b21d2f WatchSource:0}: Error finding container b797a4a79ac0c4bd1cc5741f0aab33ff2ff77338cf0caa43c2c97f7102b21d2f: Status 404 returned error can't find the container with id b797a4a79ac0c4bd1cc5741f0aab33ff2ff77338cf0caa43c2c97f7102b21d2f
May 20 15:31:34.101075 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:34.100995    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fefe56a_44d8_47d6_bae5_3926f0bc3b1c.slice/crio-e719845156638d0d27ef742420c94bace76f3f013e3fbbfb81acc7425ebbd042 WatchSource:0}: Error finding container e719845156638d0d27ef742420c94bace76f3f013e3fbbfb81acc7425ebbd042: Status 404 returned error can't find the container with id e719845156638d0d27ef742420c94bace76f3f013e3fbbfb81acc7425ebbd042
May 20 15:31:34.104330 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:34.104308    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe4adfc_e1dd_4a70_a161_f0d525f3ca3b.slice/crio-eb6da1b1c01c39a1cae38cd5d71a5c9b45c55b88f75cae222fd10701fd318740 WatchSource:0}: Error finding container eb6da1b1c01c39a1cae38cd5d71a5c9b45c55b88f75cae222fd10701fd318740: Status 404 returned error can't find the container with id eb6da1b1c01c39a1cae38cd5d71a5c9b45c55b88f75cae222fd10701fd318740
May 20 15:31:34.105663 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:34.105638    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf94e80_c129_435d_a864_29e97ca3634f.slice/crio-4b60c5539ebedc7e0bf624dffd717f804db7461d280330ed2005c7ab2545dda4 WatchSource:0}: Error finding container 4b60c5539ebedc7e0bf624dffd717f804db7461d280330ed2005c7ab2545dda4: Status 404 returned error can't find the container with id 4b60c5539ebedc7e0bf624dffd717f804db7461d280330ed2005c7ab2545dda4
May 20 15:31:34.106356 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:34.106315    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49cf359_1bf7_485e_8ed9_9171a4111b2c.slice/crio-1e6c2e357f573e0419fb8d7c5cee492dd517c2320cbe1d137a2910fd771bc127 WatchSource:0}: Error finding container 1e6c2e357f573e0419fb8d7c5cee492dd517c2320cbe1d137a2910fd771bc127: Status 404 returned error can't find the container with id 1e6c2e357f573e0419fb8d7c5cee492dd517c2320cbe1d137a2910fd771bc127
May 20 15:31:34.107521 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:34.107407    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799a84c1_8372_4b99_a34d_105048106d5d.slice/crio-3ccf08985e3c0492a1fc52e059bd964576d80f4c5d82b388d24a841552bbc2a1 WatchSource:0}: Error finding container 3ccf08985e3c0492a1fc52e059bd964576d80f4c5d82b388d24a841552bbc2a1: Status 404 returned error can't find the container with id 3ccf08985e3c0492a1fc52e059bd964576d80f4c5d82b388d24a841552bbc2a1
May 20 15:31:34.108736 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:34.108644    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5adfb93_88e3_4b63_839c_31c7a0793b04.slice/crio-b1ed1ee0dec22e35966f547f1deabc6ae22e717a95533bd40876029ab1ccfce3 WatchSource:0}: Error finding container b1ed1ee0dec22e35966f547f1deabc6ae22e717a95533bd40876029ab1ccfce3: Status 404 returned error can't find the container with id b1ed1ee0dec22e35966f547f1deabc6ae22e717a95533bd40876029ab1ccfce3
May 20 15:31:34.109079 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:34.108942    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f5b3cf_5e4f_49d9_ae81_f81badb12283.slice/crio-0fab5839f642d6be935a457078169fa4980d12f1803f280a1af65f5cc4cc5330 WatchSource:0}: Error finding container 0fab5839f642d6be935a457078169fa4980d12f1803f280a1af65f5cc4cc5330: Status 404 returned error can't find the container with id 0fab5839f642d6be935a457078169fa4980d12f1803f280a1af65f5cc4cc5330
May 20 15:31:34.256226 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:34.255974    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs\") pod \"network-metrics-daemon-skk8q\" (UID: \"3022a27a-6112-46ee-a54b-c5af5bee2a1f\") " pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:34.256389 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:34.256132    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:34.256389 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:34.256333    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs podName:3022a27a-6112-46ee-a54b-c5af5bee2a1f nodeName:}" failed. No retries permitted until 2026-05-20 15:31:35.256312307 +0000 UTC m=+4.195153856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs") pod "network-metrics-daemon-skk8q" (UID: "3022a27a-6112-46ee-a54b-c5af5bee2a1f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:34.356491 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:34.356469    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-slmsd\" (UniqueName: \"kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd\") pod \"network-check-target-r9x8t\" (UID: \"69c7e426-cc15-4efc-afbe-77377c1c3779\") " pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:31:34.356595 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:34.356583    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:31:34.356636 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:34.356597    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:31:34.356636 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:34.356605    2568 projected.go:194] Error preparing data for projected volume kube-api-access-slmsd for pod openshift-network-diagnostics/network-check-target-r9x8t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:34.356701 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:34.356647    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd podName:69c7e426-cc15-4efc-afbe-77377c1c3779 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:35.356635389 +0000 UTC m=+4.295476925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-slmsd" (UniqueName: "kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd") pod "network-check-target-r9x8t" (UID: "69c7e426-cc15-4efc-afbe-77377c1c3779") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:34.606723 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:34.606577    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 15:26:32 +0000 UTC" deadline="2028-01-12 14:34:15.282320016 +0000 UTC"
May 20 15:31:34.606723 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:34.606616    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14447h2m40.675707535s"
May 20 15:31:34.666190 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:34.666146    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal" event={"ID":"6f81a807c436a78d0ae12cc5e66b2e05","Type":"ContainerStarted","Data":"6a44b2ec0689a228dd19635056b0e53f17fdad10e51fa6a22a0ac00d99c66401"}
May 20 15:31:34.672823 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:34.672795    2568 generic.go:358] "Generic (PLEG): container finished" podID="a720ccbb8d80c2c23e6edde668812484" containerID="36440a499513ad3df138e3a596dd1355d94d6af8d0b0eae58286b525f01a930c" exitCode=0
May 20 15:31:34.673055 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:34.673035    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal" event={"ID":"a720ccbb8d80c2c23e6edde668812484","Type":"ContainerDied","Data":"36440a499513ad3df138e3a596dd1355d94d6af8d0b0eae58286b525f01a930c"}
May 20 15:31:34.683330 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:34.683286    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal" podStartSLOduration=2.683268939 podStartE2EDuration="2.683268939s" podCreationTimestamp="2026-05-20 15:31:32 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:31:34.680312927 +0000 UTC m=+3.619154486" watchObservedRunningTime="2026-05-20 15:31:34.683268939 +0000 UTC m=+3.622110497"
May 20 15:31:34.685159 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:34.685129    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bjx28" event={"ID":"d3f5b3cf-5e4f-49d9-ae81-f81badb12283","Type":"ContainerStarted","Data":"0fab5839f642d6be935a457078169fa4980d12f1803f280a1af65f5cc4cc5330"}
May 20 15:31:34.693356 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:34.693329    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g8s5k" event={"ID":"f5adfb93-88e3-4b63-839c-31c7a0793b04","Type":"ContainerStarted","Data":"b1ed1ee0dec22e35966f547f1deabc6ae22e717a95533bd40876029ab1ccfce3"}
May 20 15:31:34.698383 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:34.698340    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kffdt" event={"ID":"6fefe56a-44d8-47d6-bae5-3926f0bc3b1c","Type":"ContainerStarted","Data":"e719845156638d0d27ef742420c94bace76f3f013e3fbbfb81acc7425ebbd042"}
May 20 15:31:34.705244 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:34.705219    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-th4j4" event={"ID":"3ccfe95d-efc9-46ec-aaac-da92e41a5574","Type":"ContainerStarted","Data":"b797a4a79ac0c4bd1cc5741f0aab33ff2ff77338cf0caa43c2c97f7102b21d2f"}
May 20 15:31:34.713386 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:34.713359    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4" event={"ID":"6bf94e80-c129-435d-a864-29e97ca3634f","Type":"ContainerStarted","Data":"4b60c5539ebedc7e0bf624dffd717f804db7461d280330ed2005c7ab2545dda4"}
May 20 15:31:34.728698 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:34.728666    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvrb6" event={"ID":"799a84c1-8372-4b99-a34d-105048106d5d","Type":"ContainerStarted","Data":"3ccf08985e3c0492a1fc52e059bd964576d80f4c5d82b388d24a841552bbc2a1"}
May 20 15:31:34.733619 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:34.733525    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n" event={"ID":"b49cf359-1bf7-485e-8ed9-9171a4111b2c","Type":"ContainerStarted","Data":"1e6c2e357f573e0419fb8d7c5cee492dd517c2320cbe1d137a2910fd771bc127"}
May 20 15:31:34.747324 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:34.747294    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l24nh" event={"ID":"2fe4adfc-e1dd-4a70-a161-f0d525f3ca3b","Type":"ContainerStarted","Data":"eb6da1b1c01c39a1cae38cd5d71a5c9b45c55b88f75cae222fd10701fd318740"}
May 20 15:31:35.264345 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:35.264255    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs\") pod \"network-metrics-daemon-skk8q\" (UID: \"3022a27a-6112-46ee-a54b-c5af5bee2a1f\") " pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:35.264503 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:35.264451    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:35.264619 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:35.264520    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs podName:3022a27a-6112-46ee-a54b-c5af5bee2a1f nodeName:}" failed. No retries permitted until 2026-05-20 15:31:37.264499469 +0000 UTC m=+6.203341027 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs") pod "network-metrics-daemon-skk8q" (UID: "3022a27a-6112-46ee-a54b-c5af5bee2a1f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:35.365511 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:35.364872    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-slmsd\" (UniqueName: \"kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd\") pod \"network-check-target-r9x8t\" (UID: \"69c7e426-cc15-4efc-afbe-77377c1c3779\") " pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:31:35.365511 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:35.365069    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:31:35.365511 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:35.365090    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:31:35.365511 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:35.365102    2568 projected.go:194] Error preparing data for projected volume kube-api-access-slmsd for pod openshift-network-diagnostics/network-check-target-r9x8t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:35.365511 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:35.365161    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd podName:69c7e426-cc15-4efc-afbe-77377c1c3779 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:37.365144214 +0000 UTC m=+6.303985756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-slmsd" (UniqueName: "kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd") pod "network-check-target-r9x8t" (UID: "69c7e426-cc15-4efc-afbe-77377c1c3779") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:35.641901 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:35.641868    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:31:35.642378 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:35.641999    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9x8t" podUID="69c7e426-cc15-4efc-afbe-77377c1c3779"
May 20 15:31:35.642479 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:35.642459    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:35.642599 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:35.642576    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skk8q" podUID="3022a27a-6112-46ee-a54b-c5af5bee2a1f"
May 20 15:31:35.766160 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:35.766102    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal" event={"ID":"a720ccbb8d80c2c23e6edde668812484","Type":"ContainerStarted","Data":"0b52b308dc045bceadf521da15bb09de0556cfd195e338fc6ac093d4ab89390c"}
May 20 15:31:37.286008 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:37.285438    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs\") pod \"network-metrics-daemon-skk8q\" (UID: \"3022a27a-6112-46ee-a54b-c5af5bee2a1f\") " pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:37.286008 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:37.285615    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:37.286008 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:37.285673    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs podName:3022a27a-6112-46ee-a54b-c5af5bee2a1f nodeName:}" failed. No retries permitted until 2026-05-20 15:31:41.285655717 +0000 UTC m=+10.224497257 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs") pod "network-metrics-daemon-skk8q" (UID: "3022a27a-6112-46ee-a54b-c5af5bee2a1f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:37.386523 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:37.386485    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-slmsd\" (UniqueName: \"kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd\") pod \"network-check-target-r9x8t\" (UID: \"69c7e426-cc15-4efc-afbe-77377c1c3779\") " pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:31:37.386662 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:37.386628    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:31:37.386662 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:37.386642    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:31:37.386662 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:37.386651    2568 projected.go:194] Error preparing data for projected volume kube-api-access-slmsd for pod openshift-network-diagnostics/network-check-target-r9x8t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:37.386773 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:37.386692    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd podName:69c7e426-cc15-4efc-afbe-77377c1c3779 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:41.386680537 +0000 UTC m=+10.325522072 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-slmsd" (UniqueName: "kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd") pod "network-check-target-r9x8t" (UID: "69c7e426-cc15-4efc-afbe-77377c1c3779") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:37.639602 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:37.639168    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:37.639602 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:37.639181    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:31:37.639602 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:37.639306    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skk8q" podUID="3022a27a-6112-46ee-a54b-c5af5bee2a1f"
May 20 15:31:37.639602 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:37.639377    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9x8t" podUID="69c7e426-cc15-4efc-afbe-77377c1c3779"
May 20 15:31:39.640829 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:39.640190    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:31:39.640829 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:39.640326    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9x8t" podUID="69c7e426-cc15-4efc-afbe-77377c1c3779"
May 20 15:31:39.640829 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:39.640190    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:39.640829 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:39.640787    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skk8q" podUID="3022a27a-6112-46ee-a54b-c5af5bee2a1f"
May 20 15:31:39.891693 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:39.891602    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal" podStartSLOduration=7.891588626 podStartE2EDuration="7.891588626s" podCreationTimestamp="2026-05-20 15:31:32 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:31:35.779724955 +0000 UTC m=+4.718566514" watchObservedRunningTime="2026-05-20 15:31:39.891588626 +0000 UTC m=+8.830430184"
May 20 15:31:39.892262 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:39.892239    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-cknlk"]
May 20 15:31:39.902159 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:39.902134    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:39.902274 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:39.902224    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cknlk" podUID="cb6d0d3d-79d3-4f3f-8f8e-974108118563"
May 20 15:31:40.007770 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.007728    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb6d0d3d-79d3-4f3f-8f8e-974108118563-dbus\") pod \"global-pull-secret-syncer-cknlk\" (UID: \"cb6d0d3d-79d3-4f3f-8f8e-974108118563\") " pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:40.007933 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.007843    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb6d0d3d-79d3-4f3f-8f8e-974108118563-kubelet-config\") pod \"global-pull-secret-syncer-cknlk\" (UID: \"cb6d0d3d-79d3-4f3f-8f8e-974108118563\") " pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:40.007933 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.007890    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb6d0d3d-79d3-4f3f-8f8e-974108118563-original-pull-secret\") pod \"global-pull-secret-syncer-cknlk\" (UID: \"cb6d0d3d-79d3-4f3f-8f8e-974108118563\") " pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:40.108922 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.108881    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb6d0d3d-79d3-4f3f-8f8e-974108118563-kubelet-config\") pod \"global-pull-secret-syncer-cknlk\" (UID: \"cb6d0d3d-79d3-4f3f-8f8e-974108118563\") " pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:40.109106 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.108942    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb6d0d3d-79d3-4f3f-8f8e-974108118563-original-pull-secret\") pod \"global-pull-secret-syncer-cknlk\" (UID: \"cb6d0d3d-79d3-4f3f-8f8e-974108118563\") " pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:40.109106 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.108998    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb6d0d3d-79d3-4f3f-8f8e-974108118563-dbus\") pod \"global-pull-secret-syncer-cknlk\" (UID: \"cb6d0d3d-79d3-4f3f-8f8e-974108118563\") " pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:40.109365 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.109299    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb6d0d3d-79d3-4f3f-8f8e-974108118563-kubelet-config\") pod \"global-pull-secret-syncer-cknlk\" (UID: \"cb6d0d3d-79d3-4f3f-8f8e-974108118563\") " pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:40.109365 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.109294    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb6d0d3d-79d3-4f3f-8f8e-974108118563-dbus\") pod \"global-pull-secret-syncer-cknlk\" (UID: \"cb6d0d3d-79d3-4f3f-8f8e-974108118563\") " pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:40.109486 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:40.109403    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 15:31:40.109486 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:40.109464    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb6d0d3d-79d3-4f3f-8f8e-974108118563-original-pull-secret podName:cb6d0d3d-79d3-4f3f-8f8e-974108118563 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:40.609446356 +0000 UTC m=+9.548287896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb6d0d3d-79d3-4f3f-8f8e-974108118563-original-pull-secret") pod "global-pull-secret-syncer-cknlk" (UID: "cb6d0d3d-79d3-4f3f-8f8e-974108118563") : object "kube-system"/"original-pull-secret" not registered
May 20 15:31:40.213388 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.213309    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-vkh2n"]
May 20 15:31:40.217581 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.216800    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vkh2n"
May 20 15:31:40.221924 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.220732    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 20 15:31:40.221924 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.220958    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2bpng\""
May 20 15:31:40.221924 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.221188    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 20 15:31:40.310445 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.310229    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/abd4b634-19df-4a65-b43b-f3cd86482afe-hosts-file\") pod \"node-resolver-vkh2n\" (UID: \"abd4b634-19df-4a65-b43b-f3cd86482afe\") " pod="openshift-dns/node-resolver-vkh2n"
May 20 15:31:40.310445 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.310311    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abd4b634-19df-4a65-b43b-f3cd86482afe-tmp-dir\") pod \"node-resolver-vkh2n\" (UID: \"abd4b634-19df-4a65-b43b-f3cd86482afe\") " pod="openshift-dns/node-resolver-vkh2n"
May 20 15:31:40.310445 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.310359    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r5js\" (UniqueName: \"kubernetes.io/projected/abd4b634-19df-4a65-b43b-f3cd86482afe-kube-api-access-6r5js\") pod \"node-resolver-vkh2n\" (UID: \"abd4b634-19df-4a65-b43b-f3cd86482afe\") " pod="openshift-dns/node-resolver-vkh2n"
May 20 15:31:40.411743 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.411699    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/abd4b634-19df-4a65-b43b-f3cd86482afe-hosts-file\") pod \"node-resolver-vkh2n\" (UID: \"abd4b634-19df-4a65-b43b-f3cd86482afe\") " pod="openshift-dns/node-resolver-vkh2n"
May 20 15:31:40.411905 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.411770    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abd4b634-19df-4a65-b43b-f3cd86482afe-tmp-dir\") pod \"node-resolver-vkh2n\" (UID: \"abd4b634-19df-4a65-b43b-f3cd86482afe\") " pod="openshift-dns/node-resolver-vkh2n"
May 20 15:31:40.411905 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.411817    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/abd4b634-19df-4a65-b43b-f3cd86482afe-hosts-file\") pod \"node-resolver-vkh2n\" (UID: \"abd4b634-19df-4a65-b43b-f3cd86482afe\") " pod="openshift-dns/node-resolver-vkh2n"
May 20 15:31:40.411905 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.411843    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6r5js\" (UniqueName: \"kubernetes.io/projected/abd4b634-19df-4a65-b43b-f3cd86482afe-kube-api-access-6r5js\") pod \"node-resolver-vkh2n\" (UID: \"abd4b634-19df-4a65-b43b-f3cd86482afe\") " pod="openshift-dns/node-resolver-vkh2n"
May 20 15:31:40.412192 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.412164    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abd4b634-19df-4a65-b43b-f3cd86482afe-tmp-dir\") pod \"node-resolver-vkh2n\" (UID: \"abd4b634-19df-4a65-b43b-f3cd86482afe\") " pod="openshift-dns/node-resolver-vkh2n"
May 20 15:31:40.429051 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.428976    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r5js\" (UniqueName: \"kubernetes.io/projected/abd4b634-19df-4a65-b43b-f3cd86482afe-kube-api-access-6r5js\") pod \"node-resolver-vkh2n\" (UID: \"abd4b634-19df-4a65-b43b-f3cd86482afe\") " pod="openshift-dns/node-resolver-vkh2n"
May 20 15:31:40.528953 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.528872    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vkh2n"
May 20 15:31:40.613884 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:40.613850    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb6d0d3d-79d3-4f3f-8f8e-974108118563-original-pull-secret\") pod \"global-pull-secret-syncer-cknlk\" (UID: \"cb6d0d3d-79d3-4f3f-8f8e-974108118563\") " pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:40.614099 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:40.614083    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 15:31:40.614172 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:40.614146    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb6d0d3d-79d3-4f3f-8f8e-974108118563-original-pull-secret podName:cb6d0d3d-79d3-4f3f-8f8e-974108118563 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:41.614133093 +0000 UTC m=+10.552974628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb6d0d3d-79d3-4f3f-8f8e-974108118563-original-pull-secret") pod "global-pull-secret-syncer-cknlk" (UID: "cb6d0d3d-79d3-4f3f-8f8e-974108118563") : object "kube-system"/"original-pull-secret" not registered
May 20 15:31:41.320056 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:41.319988    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs\") pod \"network-metrics-daemon-skk8q\" (UID: \"3022a27a-6112-46ee-a54b-c5af5bee2a1f\") " pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:41.320515 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:41.320130    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:41.320515 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:41.320209    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs podName:3022a27a-6112-46ee-a54b-c5af5bee2a1f nodeName:}" failed. No retries permitted until 2026-05-20 15:31:49.320187341 +0000 UTC m=+18.259028896 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs") pod "network-metrics-daemon-skk8q" (UID: "3022a27a-6112-46ee-a54b-c5af5bee2a1f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:41.421074 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:41.421008    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-slmsd\" (UniqueName: \"kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd\") pod \"network-check-target-r9x8t\" (UID: \"69c7e426-cc15-4efc-afbe-77377c1c3779\") " pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:31:41.421259 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:41.421183    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:31:41.421259 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:41.421207    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:31:41.421259 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:41.421221    2568 projected.go:194] Error preparing data for projected volume kube-api-access-slmsd for pod openshift-network-diagnostics/network-check-target-r9x8t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:41.421419 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:41.421284    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd podName:69c7e426-cc15-4efc-afbe-77377c1c3779 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:49.421263043 +0000 UTC m=+18.360104583 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-slmsd" (UniqueName: "kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd") pod "network-check-target-r9x8t" (UID: "69c7e426-cc15-4efc-afbe-77377c1c3779") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:41.622694 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:41.622608    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb6d0d3d-79d3-4f3f-8f8e-974108118563-original-pull-secret\") pod \"global-pull-secret-syncer-cknlk\" (UID: \"cb6d0d3d-79d3-4f3f-8f8e-974108118563\") " pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:41.622832 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:41.622775    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 15:31:41.622869 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:41.622850    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb6d0d3d-79d3-4f3f-8f8e-974108118563-original-pull-secret podName:cb6d0d3d-79d3-4f3f-8f8e-974108118563 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:43.622837445 +0000 UTC m=+12.561678981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb6d0d3d-79d3-4f3f-8f8e-974108118563-original-pull-secret") pod "global-pull-secret-syncer-cknlk" (UID: "cb6d0d3d-79d3-4f3f-8f8e-974108118563") : object "kube-system"/"original-pull-secret" not registered
May 20 15:31:41.640558 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:41.640525    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:41.640719 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:41.640573    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:31:41.640719 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:41.640524    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:41.640719 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:41.640698    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skk8q" podUID="3022a27a-6112-46ee-a54b-c5af5bee2a1f"
May 20 15:31:41.640884 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:41.640796    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cknlk" podUID="cb6d0d3d-79d3-4f3f-8f8e-974108118563"
May 20 15:31:41.640937 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:41.640887    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9x8t" podUID="69c7e426-cc15-4efc-afbe-77377c1c3779"
May 20 15:31:43.639551 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:43.639037    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:43.639551 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:43.639081    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:43.639551 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:43.639142    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb6d0d3d-79d3-4f3f-8f8e-974108118563-original-pull-secret\") pod \"global-pull-secret-syncer-cknlk\" (UID: \"cb6d0d3d-79d3-4f3f-8f8e-974108118563\") " pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:43.639551 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:43.639183    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skk8q" podUID="3022a27a-6112-46ee-a54b-c5af5bee2a1f"
May 20 15:31:43.639551 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:43.639250    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 15:31:43.639551 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:43.639312    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb6d0d3d-79d3-4f3f-8f8e-974108118563-original-pull-secret podName:cb6d0d3d-79d3-4f3f-8f8e-974108118563 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:47.639291396 +0000 UTC m=+16.578132947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb6d0d3d-79d3-4f3f-8f8e-974108118563-original-pull-secret") pod "global-pull-secret-syncer-cknlk" (UID: "cb6d0d3d-79d3-4f3f-8f8e-974108118563") : object "kube-system"/"original-pull-secret" not registered
May 20 15:31:43.639551 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:43.639381    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:31:43.639551 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:43.639373    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cknlk" podUID="cb6d0d3d-79d3-4f3f-8f8e-974108118563"
May 20 15:31:43.639551 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:43.639472    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9x8t" podUID="69c7e426-cc15-4efc-afbe-77377c1c3779"
May 20 15:31:45.639033 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:45.638975    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:31:45.639033 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:45.638980    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:45.639566 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:45.638981    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:45.639566 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:45.639145    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9x8t" podUID="69c7e426-cc15-4efc-afbe-77377c1c3779"
May 20 15:31:45.639566 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:45.639239    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skk8q" podUID="3022a27a-6112-46ee-a54b-c5af5bee2a1f"
May 20 15:31:45.639566 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:45.639329    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cknlk" podUID="cb6d0d3d-79d3-4f3f-8f8e-974108118563"
May 20 15:31:47.638775 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:47.638732    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:31:47.639263 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:47.638732    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:47.639263 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:47.638863    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9x8t" podUID="69c7e426-cc15-4efc-afbe-77377c1c3779"
May 20 15:31:47.639263 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:47.638746    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:47.639263 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:47.638938    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cknlk" podUID="cb6d0d3d-79d3-4f3f-8f8e-974108118563"
May 20 15:31:47.639263 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:47.639074    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skk8q" podUID="3022a27a-6112-46ee-a54b-c5af5bee2a1f"
May 20 15:31:47.670596 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:47.670558    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb6d0d3d-79d3-4f3f-8f8e-974108118563-original-pull-secret\") pod \"global-pull-secret-syncer-cknlk\" (UID: \"cb6d0d3d-79d3-4f3f-8f8e-974108118563\") " pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:47.670763 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:47.670679    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 15:31:47.670763 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:47.670745    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb6d0d3d-79d3-4f3f-8f8e-974108118563-original-pull-secret podName:cb6d0d3d-79d3-4f3f-8f8e-974108118563 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:55.670724478 +0000 UTC m=+24.609566013 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb6d0d3d-79d3-4f3f-8f8e-974108118563-original-pull-secret") pod "global-pull-secret-syncer-cknlk" (UID: "cb6d0d3d-79d3-4f3f-8f8e-974108118563") : object "kube-system"/"original-pull-secret" not registered
May 20 15:31:49.383212 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:49.382967    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs\") pod \"network-metrics-daemon-skk8q\" (UID: \"3022a27a-6112-46ee-a54b-c5af5bee2a1f\") " pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:49.383642 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:49.383135    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:49.383642 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:49.383349    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs podName:3022a27a-6112-46ee-a54b-c5af5bee2a1f nodeName:}" failed. No retries permitted until 2026-05-20 15:32:05.383329059 +0000 UTC m=+34.322170614 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs") pod "network-metrics-daemon-skk8q" (UID: "3022a27a-6112-46ee-a54b-c5af5bee2a1f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:49.484370 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:49.484337    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-slmsd\" (UniqueName: \"kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd\") pod \"network-check-target-r9x8t\" (UID: \"69c7e426-cc15-4efc-afbe-77377c1c3779\") " pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:31:49.484532 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:49.484520    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:31:49.484574 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:49.484539    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:31:49.484574 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:49.484548    2568 projected.go:194] Error preparing data for projected volume kube-api-access-slmsd for pod openshift-network-diagnostics/network-check-target-r9x8t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:49.484666 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:49.484600    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd podName:69c7e426-cc15-4efc-afbe-77377c1c3779 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:05.484587656 +0000 UTC m=+34.423429191 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-slmsd" (UniqueName: "kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd") pod "network-check-target-r9x8t" (UID: "69c7e426-cc15-4efc-afbe-77377c1c3779") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:49.639256 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:49.639174    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:49.639399 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:49.639322    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skk8q" podUID="3022a27a-6112-46ee-a54b-c5af5bee2a1f"
May 20 15:31:49.639399 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:49.639323    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:31:49.639399 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:49.639360    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:49.639542 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:49.639435    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9x8t" podUID="69c7e426-cc15-4efc-afbe-77377c1c3779"
May 20 15:31:49.639542 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:49.639506    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cknlk" podUID="cb6d0d3d-79d3-4f3f-8f8e-974108118563"
May 20 15:31:51.640741 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:51.640710    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:51.641316 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:51.640780    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:31:51.641316 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:51.640799    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:51.641316 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:51.640888    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9x8t" podUID="69c7e426-cc15-4efc-afbe-77377c1c3779"
May 20 15:31:51.641316 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:51.640950    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skk8q" podUID="3022a27a-6112-46ee-a54b-c5af5bee2a1f"
May 20 15:31:51.641316 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:51.641011    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cknlk" podUID="cb6d0d3d-79d3-4f3f-8f8e-974108118563"
May 20 15:31:52.757472 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:31:52.757438    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd4b634_19df_4a65_b43b_f3cd86482afe.slice/crio-e7a47ff7624c271e31dd1cd5b01f6e53b00f1f3ead9b91462f38ccffae26afdc WatchSource:0}: Error finding container e7a47ff7624c271e31dd1cd5b01f6e53b00f1f3ead9b91462f38ccffae26afdc: Status 404 returned error can't find the container with id e7a47ff7624c271e31dd1cd5b01f6e53b00f1f3ead9b91462f38ccffae26afdc
May 20 15:31:52.797986 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:52.797946    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vkh2n" event={"ID":"abd4b634-19df-4a65-b43b-f3cd86482afe","Type":"ContainerStarted","Data":"e7a47ff7624c271e31dd1cd5b01f6e53b00f1f3ead9b91462f38ccffae26afdc"}
May 20 15:31:53.640704 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:53.638908    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:53.640704 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:53.639054    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skk8q" podUID="3022a27a-6112-46ee-a54b-c5af5bee2a1f"
May 20 15:31:53.640704 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:53.639148    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:31:53.640704 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:53.639249    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9x8t" podUID="69c7e426-cc15-4efc-afbe-77377c1c3779"
May 20 15:31:53.640704 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:53.639295    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:53.640704 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:53.639360    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cknlk" podUID="cb6d0d3d-79d3-4f3f-8f8e-974108118563"
May 20 15:31:53.801145 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:53.801062    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-th4j4" event={"ID":"3ccfe95d-efc9-46ec-aaac-da92e41a5574","Type":"ContainerStarted","Data":"0f49939f901526e4d4cfb9b2e2baf4f44cd5b289aea3ec0aa4da7a5abc8cb049"}
May 20 15:31:53.802444 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:53.802426    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 15:31:53.802690 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:53.802670    2568 generic.go:358] "Generic (PLEG): container finished" podID="6bf94e80-c129-435d-a864-29e97ca3634f" containerID="d540cdff12fb1f8450555bd6d6672ea5111d4fa9c6ca8c07bbd61c690fbfc297" exitCode=1
May 20 15:31:53.802763 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:53.802740    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4" event={"ID":"6bf94e80-c129-435d-a864-29e97ca3634f","Type":"ContainerDied","Data":"d540cdff12fb1f8450555bd6d6672ea5111d4fa9c6ca8c07bbd61c690fbfc297"}
May 20 15:31:53.802804 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:53.802767    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4" event={"ID":"6bf94e80-c129-435d-a864-29e97ca3634f","Type":"ContainerStarted","Data":"a515da1561c1a25731852bb38c7a117eef7a5259d736f14569c474f7c26f815a"}
May 20 15:31:53.803881 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:53.803860    2568 generic.go:358] "Generic (PLEG): container finished" podID="799a84c1-8372-4b99-a34d-105048106d5d" containerID="eec9fd530d54f4806e050e7e546a6c60bed7e8f999c7de87c0929f19e022f0e9" exitCode=0
May 20 15:31:53.803935 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:53.803915    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvrb6" event={"ID":"799a84c1-8372-4b99-a34d-105048106d5d","Type":"ContainerDied","Data":"eec9fd530d54f4806e050e7e546a6c60bed7e8f999c7de87c0929f19e022f0e9"}
May 20 15:31:53.805297 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:53.805261    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n" event={"ID":"b49cf359-1bf7-485e-8ed9-9171a4111b2c","Type":"ContainerStarted","Data":"5bb3f274d86a2df4824c8e852ead94c3baae971d6737799162f3802972cf9eb6"}
May 20 15:31:53.806489 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:53.806460    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l24nh" event={"ID":"2fe4adfc-e1dd-4a70-a161-f0d525f3ca3b","Type":"ContainerStarted","Data":"021443a7cae04e5d6842663a89acfc6a15bc92f7e6febd0835e47e357c41e301"}
May 20 15:31:53.807698 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:53.807680    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vkh2n" event={"ID":"abd4b634-19df-4a65-b43b-f3cd86482afe","Type":"ContainerStarted","Data":"57efe205a3789522062f32759eee0258b1aec3a94a4d7d42f193bf5bb8cf05ec"}
May 20 15:31:53.808809 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:53.808790    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bjx28" event={"ID":"d3f5b3cf-5e4f-49d9-ae81-f81badb12283","Type":"ContainerStarted","Data":"ac6db83f2375410afdc5e8997ff46cb2cef5a59ce9b6c2f5330573023c98afef"}
May 20 15:31:53.809888 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:53.809870    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g8s5k" event={"ID":"f5adfb93-88e3-4b63-839c-31c7a0793b04","Type":"ContainerStarted","Data":"4755d4aca1d3f0fdb004a50f75ab5961acd686d0bd095a915516ddf97d028416"}
May 20 15:31:53.818877 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:53.818842    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-th4j4" podStartSLOduration=4.110986646 podStartE2EDuration="22.81881589s" podCreationTimestamp="2026-05-20 15:31:31 +0000 UTC" firstStartedPulling="2026-05-20 15:31:34.101994415 +0000 UTC m=+3.040835951" lastFinishedPulling="2026-05-20 15:31:52.809823659 +0000 UTC m=+21.748665195" observedRunningTime="2026-05-20 15:31:53.81878095 +0000 UTC m=+22.757622507" watchObservedRunningTime="2026-05-20 15:31:53.81881589 +0000 UTC m=+22.757657447"
May 20 15:31:53.836321 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:53.836278    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-vkh2n" podStartSLOduration=13.836266694 podStartE2EDuration="13.836266694s" podCreationTimestamp="2026-05-20 15:31:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:31:53.835756122 +0000 UTC m=+22.774597679" watchObservedRunningTime="2026-05-20 15:31:53.836266694 +0000 UTC m=+22.775108251"
May 20 15:31:53.853062 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:53.852974    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g8s5k" podStartSLOduration=8.947644704 podStartE2EDuration="22.852949456s" podCreationTimestamp="2026-05-20 15:31:31 +0000 UTC" firstStartedPulling="2026-05-20 15:31:34.110702347 +0000 UTC m=+3.049543895" lastFinishedPulling="2026-05-20 15:31:48.016007111 +0000 UTC m=+16.954848647" observedRunningTime="2026-05-20 15:31:53.852525739 +0000 UTC m=+22.791367291" watchObservedRunningTime="2026-05-20 15:31:53.852949456 +0000 UTC m=+22.791791015"
May 20 15:31:53.867181 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:53.867142    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-l24nh" podStartSLOduration=4.190907376 podStartE2EDuration="22.867128431s" podCreationTimestamp="2026-05-20 15:31:31 +0000 UTC" firstStartedPulling="2026-05-20 15:31:34.10614936 +0000 UTC m=+3.044990914" lastFinishedPulling="2026-05-20 15:31:52.78237043 +0000 UTC m=+21.721211969" observedRunningTime="2026-05-20 15:31:53.86679528 +0000 UTC m=+22.805636831" watchObservedRunningTime="2026-05-20 15:31:53.867128431 +0000 UTC m=+22.805969988"
May 20 15:31:53.904124 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:53.904080    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-bjx28" podStartSLOduration=4.201835561 podStartE2EDuration="22.904065092s" podCreationTimestamp="2026-05-20 15:31:31 +0000 UTC" firstStartedPulling="2026-05-20 15:31:34.111619812 +0000 UTC m=+3.050461350" lastFinishedPulling="2026-05-20 15:31:52.813849346 +0000 UTC m=+21.752690881" observedRunningTime="2026-05-20 15:31:53.903798325 +0000 UTC m=+22.842639884" watchObservedRunningTime="2026-05-20 15:31:53.904065092 +0000 UTC m=+22.842906649"
May 20 15:31:54.432667 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:54.432469    2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 20 15:31:54.623165 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:54.622986    2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-20T15:31:54.432668031Z","UUID":"1b11409f-6aa5-4542-bbd8-3258ed0e0568","Handler":null,"Name":"","Endpoint":""}
May 20 15:31:54.626508 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:54.626481    2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 20 15:31:54.626508 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:54.626514    2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 20 15:31:54.816826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:54.816797    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 15:31:54.823200 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:54.823163    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4" event={"ID":"6bf94e80-c129-435d-a864-29e97ca3634f","Type":"ContainerStarted","Data":"7dd922e0f832ec0bf2b355ff78147c4a1009f90b144f84c58a119b2a2f5eea64"}
May 20 15:31:54.823368 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:54.823207    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4" event={"ID":"6bf94e80-c129-435d-a864-29e97ca3634f","Type":"ContainerStarted","Data":"d41aa6e1b2de7586b3fdfc921e1fe1b08847f7482094a5d1c7bbe8adb1e6f2a2"}
May 20 15:31:54.823368 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:54.823220    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4" event={"ID":"6bf94e80-c129-435d-a864-29e97ca3634f","Type":"ContainerStarted","Data":"0d40dbafaff11f9cb9a54b45b9de3fb1f763a4c18e014dc3b40d8a841db8af17"}
May 20 15:31:54.823368 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:54.823229    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4" event={"ID":"6bf94e80-c129-435d-a864-29e97ca3634f","Type":"ContainerStarted","Data":"05e5f7468ec74709e455be9b87275cad756daef4e3bc53d859e7a46a35544617"}
May 20 15:31:54.824952 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:54.824918    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n" event={"ID":"b49cf359-1bf7-485e-8ed9-9171a4111b2c","Type":"ContainerStarted","Data":"a3fcf409d5af610131e03777d597673952b1028a42d547626d1ffbc811f3cfe9"}
May 20 15:31:54.826474 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:54.826445    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kffdt" event={"ID":"6fefe56a-44d8-47d6-bae5-3926f0bc3b1c","Type":"ContainerStarted","Data":"57f362b92f2848769e81c130ae5f88bdd06c7b108ef116028479a12445d9964b"}
May 20 15:31:54.841272 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:54.841207    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kffdt" podStartSLOduration=13.395873593 podStartE2EDuration="23.841190445s" podCreationTimestamp="2026-05-20 15:31:31 +0000 UTC" firstStartedPulling="2026-05-20 15:31:34.10323268 +0000 UTC m=+3.042074215" lastFinishedPulling="2026-05-20 15:31:44.548549523 +0000 UTC m=+13.487391067" observedRunningTime="2026-05-20 15:31:54.840611932 +0000 UTC m=+23.779453488" watchObservedRunningTime="2026-05-20 15:31:54.841190445 +0000 UTC m=+23.780032006"
May 20 15:31:55.639196 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:55.639164    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:55.639371 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:55.639164    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:31:55.639371 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:55.639281    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cknlk" podUID="cb6d0d3d-79d3-4f3f-8f8e-974108118563"
May 20 15:31:55.639371 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:55.639177    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:55.639531 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:55.639376    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9x8t" podUID="69c7e426-cc15-4efc-afbe-77377c1c3779"
May 20 15:31:55.639531 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:55.639466    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skk8q" podUID="3022a27a-6112-46ee-a54b-c5af5bee2a1f"
May 20 15:31:55.732331 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:55.732292    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb6d0d3d-79d3-4f3f-8f8e-974108118563-original-pull-secret\") pod \"global-pull-secret-syncer-cknlk\" (UID: \"cb6d0d3d-79d3-4f3f-8f8e-974108118563\") " pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:55.732520 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:55.732464    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 15:31:55.732579 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:55.732533    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb6d0d3d-79d3-4f3f-8f8e-974108118563-original-pull-secret podName:cb6d0d3d-79d3-4f3f-8f8e-974108118563 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:11.732516069 +0000 UTC m=+40.671357604 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb6d0d3d-79d3-4f3f-8f8e-974108118563-original-pull-secret") pod "global-pull-secret-syncer-cknlk" (UID: "cb6d0d3d-79d3-4f3f-8f8e-974108118563") : object "kube-system"/"original-pull-secret" not registered
May 20 15:31:55.830369 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:55.830337    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n" event={"ID":"b49cf359-1bf7-485e-8ed9-9171a4111b2c","Type":"ContainerStarted","Data":"5881d98e71b1fe049542c455e9d7e52967f1fc92ef6a6cf2a0e30211b4d5c12a"}
May 20 15:31:55.850436 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:55.850388    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzb4n" podStartSLOduration=3.466800087 podStartE2EDuration="24.850373052s" podCreationTimestamp="2026-05-20 15:31:31 +0000 UTC" firstStartedPulling="2026-05-20 15:31:34.108526714 +0000 UTC m=+3.047368249" lastFinishedPulling="2026-05-20 15:31:55.492099668 +0000 UTC m=+24.430941214" observedRunningTime="2026-05-20 15:31:55.849871673 +0000 UTC m=+24.788713231" watchObservedRunningTime="2026-05-20 15:31:55.850373052 +0000 UTC m=+24.789214608"
May 20 15:31:57.639207 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:57.639167    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:57.639678 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:57.639167    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:57.639678 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:57.639309    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cknlk" podUID="cb6d0d3d-79d3-4f3f-8f8e-974108118563"
May 20 15:31:57.639678 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:57.639167    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:31:57.639678 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:57.639365    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skk8q" podUID="3022a27a-6112-46ee-a54b-c5af5bee2a1f"
May 20 15:31:57.639678 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:57.639460    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9x8t" podUID="69c7e426-cc15-4efc-afbe-77377c1c3779"
May 20 15:31:57.809878 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:57.809841    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g8s5k"
May 20 15:31:57.810490 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:57.810470    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g8s5k"
May 20 15:31:57.834789 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:57.834555    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g8s5k"
May 20 15:31:57.835087 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:57.835060    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g8s5k"
May 20 15:31:58.838184 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:58.838157    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 15:31:58.838581 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:58.838477    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4" event={"ID":"6bf94e80-c129-435d-a864-29e97ca3634f","Type":"ContainerStarted","Data":"d36c8c553ceb8f426a4f88b1af2f140716544b0043eb4ea080344fa95f5ae1d5"}
May 20 15:31:58.839876 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:58.839851    2568 generic.go:358] "Generic (PLEG): container finished" podID="799a84c1-8372-4b99-a34d-105048106d5d" containerID="2259578e1d8bf14886f444ec203a53bc763704ac251dfbac8f26ab40287eab8f" exitCode=0
May 20 15:31:58.840003 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:58.839929    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvrb6" event={"ID":"799a84c1-8372-4b99-a34d-105048106d5d","Type":"ContainerDied","Data":"2259578e1d8bf14886f444ec203a53bc763704ac251dfbac8f26ab40287eab8f"}
May 20 15:31:59.639100 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:59.639078    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:31:59.639204 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:59.639118    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:31:59.639204 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:59.639077    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:31:59.639204 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:59.639191    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skk8q" podUID="3022a27a-6112-46ee-a54b-c5af5bee2a1f"
May 20 15:31:59.639361 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:59.639264    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cknlk" podUID="cb6d0d3d-79d3-4f3f-8f8e-974108118563"
May 20 15:31:59.639361 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:31:59.639334    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9x8t" podUID="69c7e426-cc15-4efc-afbe-77377c1c3779"
May 20 15:31:59.843343 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:59.843305    2568 generic.go:358] "Generic (PLEG): container finished" podID="799a84c1-8372-4b99-a34d-105048106d5d" containerID="58faf37b888aa77cadd609db37d4216e52f539a0a8341f0e2d81dedf059acda0" exitCode=0
May 20 15:31:59.843777 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:31:59.843349    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvrb6" event={"ID":"799a84c1-8372-4b99-a34d-105048106d5d","Type":"ContainerDied","Data":"58faf37b888aa77cadd609db37d4216e52f539a0a8341f0e2d81dedf059acda0"}
May 20 15:32:00.848470 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:00.848302    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 15:32:00.849164 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:00.848761    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4" event={"ID":"6bf94e80-c129-435d-a864-29e97ca3634f","Type":"ContainerStarted","Data":"42fe40ec3e6489a6c32a7c8edcadf7c13f90348f836a17b0b21089038351e36c"}
May 20 15:32:00.849164 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:00.849094    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:32:00.849315 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:00.849291    2568 scope.go:117] "RemoveContainer" containerID="d540cdff12fb1f8450555bd6d6672ea5111d4fa9c6ca8c07bbd61c690fbfc297"
May 20 15:32:00.851235 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:00.851209    2568 generic.go:358] "Generic (PLEG): container finished" podID="799a84c1-8372-4b99-a34d-105048106d5d" containerID="e2d03b07ed001b34e27c76c670ed8f2f128a30b9e408c5b17be50bc69bfc49be" exitCode=0
May 20 15:32:00.851346 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:00.851247    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvrb6" event={"ID":"799a84c1-8372-4b99-a34d-105048106d5d","Type":"ContainerDied","Data":"e2d03b07ed001b34e27c76c670ed8f2f128a30b9e408c5b17be50bc69bfc49be"}
May 20 15:32:00.866161 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:00.866112    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:32:01.642966 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:01.642936    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:32:01.643164 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:01.643074    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cknlk" podUID="cb6d0d3d-79d3-4f3f-8f8e-974108118563"
May 20 15:32:01.643164 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:01.643122    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:32:01.643270 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:01.643175    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:32:01.643338 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:01.643277    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skk8q" podUID="3022a27a-6112-46ee-a54b-c5af5bee2a1f"
May 20 15:32:01.643390 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:01.643343    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9x8t" podUID="69c7e426-cc15-4efc-afbe-77377c1c3779"
May 20 15:32:01.862054 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:01.860704    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 15:32:01.862054 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:01.861162    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4" event={"ID":"6bf94e80-c129-435d-a864-29e97ca3634f","Type":"ContainerStarted","Data":"cfd828d42cd91d617f737bfa445f6f93ecd6d185edd4b37c3ae7670c0d5ed715"}
May 20 15:32:01.862054 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:01.861613    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:32:01.862054 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:01.861650    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:32:01.880551 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:01.880524    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:32:01.904552 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:01.902117    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4" podStartSLOduration=11.618003001 podStartE2EDuration="30.90209562s" podCreationTimestamp="2026-05-20 15:31:31 +0000 UTC" firstStartedPulling="2026-05-20 15:31:34.108034828 +0000 UTC m=+3.046876363" lastFinishedPulling="2026-05-20 15:31:53.392127433 +0000 UTC m=+22.330968982" observedRunningTime="2026-05-20 15:32:01.901519288 +0000 UTC m=+30.840360846" watchObservedRunningTime="2026-05-20 15:32:01.90209562 +0000 UTC m=+30.840937179"
May 20 15:32:02.077257 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:02.077225    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cknlk"]
May 20 15:32:02.077419 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:02.077368    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:32:02.077495 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:02.077465    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cknlk" podUID="cb6d0d3d-79d3-4f3f-8f8e-974108118563"
May 20 15:32:02.080348 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:02.080319    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r9x8t"]
May 20 15:32:02.080490 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:02.080431    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:32:02.080552 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:02.080514    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9x8t" podUID="69c7e426-cc15-4efc-afbe-77377c1c3779"
May 20 15:32:02.083277 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:02.083249    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-skk8q"]
May 20 15:32:02.083378 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:02.083369    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:32:02.083501 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:02.083479    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skk8q" podUID="3022a27a-6112-46ee-a54b-c5af5bee2a1f"
May 20 15:32:03.642169 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:03.642138    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:32:03.642632 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:03.642182    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:32:03.642632 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:03.642234    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:32:03.642632 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:03.642377    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cknlk" podUID="cb6d0d3d-79d3-4f3f-8f8e-974108118563"
May 20 15:32:03.642632 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:03.642437    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9x8t" podUID="69c7e426-cc15-4efc-afbe-77377c1c3779"
May 20 15:32:03.642632 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:03.642536    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skk8q" podUID="3022a27a-6112-46ee-a54b-c5af5bee2a1f"
May 20 15:32:05.404248 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.404212    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs\") pod \"network-metrics-daemon-skk8q\" (UID: \"3022a27a-6112-46ee-a54b-c5af5bee2a1f\") " pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:32:05.404712 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:05.404416    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:05.404712 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:05.404501    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs podName:3022a27a-6112-46ee-a54b-c5af5bee2a1f nodeName:}" failed. No retries permitted until 2026-05-20 15:32:37.404478595 +0000 UTC m=+66.343320134 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs") pod "network-metrics-daemon-skk8q" (UID: "3022a27a-6112-46ee-a54b-c5af5bee2a1f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:05.504868 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.504831    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-slmsd\" (UniqueName: \"kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd\") pod \"network-check-target-r9x8t\" (UID: \"69c7e426-cc15-4efc-afbe-77377c1c3779\") " pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:32:05.505090 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:05.505069    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:32:05.505166 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:05.505100    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:32:05.505166 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:05.505113    2568 projected.go:194] Error preparing data for projected volume kube-api-access-slmsd for pod openshift-network-diagnostics/network-check-target-r9x8t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:05.505283 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:05.505179    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd podName:69c7e426-cc15-4efc-afbe-77377c1c3779 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:37.505160504 +0000 UTC m=+66.444002056 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-slmsd" (UniqueName: "kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd") pod "network-check-target-r9x8t" (UID: "69c7e426-cc15-4efc-afbe-77377c1c3779") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:05.639037 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.638978    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:32:05.639210 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.638978    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:32:05.639210 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:05.639127    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r9x8t" podUID="69c7e426-cc15-4efc-afbe-77377c1c3779"
May 20 15:32:05.639210 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.638978    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:32:05.639366 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:05.639208    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skk8q" podUID="3022a27a-6112-46ee-a54b-c5af5bee2a1f"
May 20 15:32:05.639366 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:05.639322    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cknlk" podUID="cb6d0d3d-79d3-4f3f-8f8e-974108118563"
May 20 15:32:05.840112 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.840079    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeReady"
May 20 15:32:05.840303 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.840236    2568 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 20 15:32:05.873554 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.873528    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj"]
May 20 15:32:05.904082 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.904056    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-666497c696-xxwnj"]
May 20 15:32:05.904270 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.904236    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj"
May 20 15:32:05.906766 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.906738    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
May 20 15:32:05.906885 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.906786    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
May 20 15:32:05.906955 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.906943    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ffrjz\""
May 20 15:32:05.928261 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.928235    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w8bsv"]
May 20 15:32:05.928403 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.928387    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:05.931570 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.931548    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 20 15:32:05.931712 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.931689    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 20 15:32:05.931804 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.931791    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 20 15:32:05.931878 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.931862    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9frfm\""
May 20 15:32:05.937332 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.937313    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 20 15:32:05.951048 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.950911    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj"]
May 20 15:32:05.951048 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.950943    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-666497c696-xxwnj"]
May 20 15:32:05.951048 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.950954    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w8bsv"]
May 20 15:32:05.951048 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.950967    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-szz5l"]
May 20 15:32:05.951566 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.951378    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w8bsv"
May 20 15:32:05.953509 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.953492    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-trd7m\""
May 20 15:32:05.954165 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.954146    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 20 15:32:05.954259 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.954146    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 20 15:32:05.976061 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.976033    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-szz5l"]
May 20 15:32:05.976196 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.976064    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-szz5l"
May 20 15:32:05.978828 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.978798    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 20 15:32:05.978937 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.978806    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 20 15:32:05.978937 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.978861    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 20 15:32:05.978937 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:05.978934    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4srs2\""
May 20 15:32:06.008608 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.008573    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c010265c-2cd5-4e4f-9fd5-f75755dc1419-ca-trust-extracted\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.008771 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.008621    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c010265c-2cd5-4e4f-9fd5-f75755dc1419-installation-pull-secrets\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.008771 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.008645    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-2fdvj\" (UID: \"94373970-3fe5-4bbf-a05a-f4913b27e8d0\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj"
May 20 15:32:06.008771 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.008671    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8cbs\" (UniqueName: \"kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-kube-api-access-h8cbs\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.008771 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.008710    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.008771 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.008732    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/94373970-3fe5-4bbf-a05a-f4913b27e8d0-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-2fdvj\" (UID: \"94373970-3fe5-4bbf-a05a-f4913b27e8d0\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj"
May 20 15:32:06.009060 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.008824    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-certificates\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.009060 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.008854    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c010265c-2cd5-4e4f-9fd5-f75755dc1419-trusted-ca\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.009060 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.008871    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-bound-sa-token\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.009060 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.008947    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c010265c-2cd5-4e4f-9fd5-f75755dc1419-image-registry-private-configuration\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.110125 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.110042    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c010265c-2cd5-4e4f-9fd5-f75755dc1419-ca-trust-extracted\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.110125 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.110091    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c010265c-2cd5-4e4f-9fd5-f75755dc1419-installation-pull-secrets\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.110125 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.110115    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-2fdvj\" (UID: \"94373970-3fe5-4bbf-a05a-f4913b27e8d0\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj"
May 20 15:32:06.110410 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.110143    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h8cbs\" (UniqueName: \"kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-kube-api-access-h8cbs\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.110410 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.110172    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9bcdce67-2f01-4c5e-a20f-c976159f8683-config-volume\") pod \"dns-default-w8bsv\" (UID: \"9bcdce67-2f01-4c5e-a20f-c976159f8683\") " pod="openshift-dns/dns-default-w8bsv"
May 20 15:32:06.110410 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.110197    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bcdce67-2f01-4c5e-a20f-c976159f8683-tmp-dir\") pod \"dns-default-w8bsv\" (UID: \"9bcdce67-2f01-4c5e-a20f-c976159f8683\") " pod="openshift-dns/dns-default-w8bsv"
May 20 15:32:06.110410 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.110227    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.110410 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:06.110250    2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 20 15:32:06.110410 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:06.110323    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert podName:94373970-3fe5-4bbf-a05a-f4913b27e8d0 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:06.610301697 +0000 UTC m=+35.549143233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-2fdvj" (UID: "94373970-3fe5-4bbf-a05a-f4913b27e8d0") : secret "networking-console-plugin-cert" not found
May 20 15:32:06.110410 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:06.110333    2568 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 15:32:06.110410 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:06.110351    2568 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666497c696-xxwnj: secret "image-registry-tls" not found
May 20 15:32:06.115213 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.115187    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c010265c-2cd5-4e4f-9fd5-f75755dc1419-ca-trust-extracted\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.115337 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.110252    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/94373970-3fe5-4bbf-a05a-f4913b27e8d0-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-2fdvj\" (UID: \"94373970-3fe5-4bbf-a05a-f4913b27e8d0\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj"
May 20 15:32:06.115337 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:06.115278    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls podName:c010265c-2cd5-4e4f-9fd5-f75755dc1419 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:06.615248913 +0000 UTC m=+35.554090447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls") pod "image-registry-666497c696-xxwnj" (UID: "c010265c-2cd5-4e4f-9fd5-f75755dc1419") : secret "image-registry-tls" not found
May 20 15:32:06.115453 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.115338    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-certificates\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.115453 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.115375    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c010265c-2cd5-4e4f-9fd5-f75755dc1419-trusted-ca\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.115453 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.115407    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-bound-sa-token\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.115590 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.115454    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c010265c-2cd5-4e4f-9fd5-f75755dc1419-image-registry-private-configuration\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.115590 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.115516    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls\") pod \"dns-default-w8bsv\" (UID: \"9bcdce67-2f01-4c5e-a20f-c976159f8683\") " pod="openshift-dns/dns-default-w8bsv"
May 20 15:32:06.115590 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.115546    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbhtr\" (UniqueName: \"kubernetes.io/projected/9bcdce67-2f01-4c5e-a20f-c976159f8683-kube-api-access-dbhtr\") pod \"dns-default-w8bsv\" (UID: \"9bcdce67-2f01-4c5e-a20f-c976159f8683\") " pod="openshift-dns/dns-default-w8bsv"
May 20 15:32:06.115590 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.115578    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert\") pod \"ingress-canary-szz5l\" (UID: \"fbac1f01-44d4-4042-a375-5a6a14f94d47\") " pod="openshift-ingress-canary/ingress-canary-szz5l"
May 20 15:32:06.115755 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.115617    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwn4h\" (UniqueName: \"kubernetes.io/projected/fbac1f01-44d4-4042-a375-5a6a14f94d47-kube-api-access-nwn4h\") pod \"ingress-canary-szz5l\" (UID: \"fbac1f01-44d4-4042-a375-5a6a14f94d47\") " pod="openshift-ingress-canary/ingress-canary-szz5l"
May 20 15:32:06.115755 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.115709    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/94373970-3fe5-4bbf-a05a-f4913b27e8d0-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-2fdvj\" (UID: \"94373970-3fe5-4bbf-a05a-f4913b27e8d0\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj"
May 20 15:32:06.116047 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.116008    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-certificates\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.116666 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.116639    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c010265c-2cd5-4e4f-9fd5-f75755dc1419-trusted-ca\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.119881 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.119858    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c010265c-2cd5-4e4f-9fd5-f75755dc1419-image-registry-private-configuration\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.119979 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.119862    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c010265c-2cd5-4e4f-9fd5-f75755dc1419-installation-pull-secrets\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.124372 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.124117    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-bound-sa-token\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.125584 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.125561    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8cbs\" (UniqueName: \"kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-kube-api-access-h8cbs\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.216739 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.216704    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9bcdce67-2f01-4c5e-a20f-c976159f8683-config-volume\") pod \"dns-default-w8bsv\" (UID: \"9bcdce67-2f01-4c5e-a20f-c976159f8683\") " pod="openshift-dns/dns-default-w8bsv"
May 20 15:32:06.216739 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.216742    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bcdce67-2f01-4c5e-a20f-c976159f8683-tmp-dir\") pod \"dns-default-w8bsv\" (UID: \"9bcdce67-2f01-4c5e-a20f-c976159f8683\") " pod="openshift-dns/dns-default-w8bsv"
May 20 15:32:06.216978 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.216825    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls\") pod \"dns-default-w8bsv\" (UID: \"9bcdce67-2f01-4c5e-a20f-c976159f8683\") " pod="openshift-dns/dns-default-w8bsv"
May 20 15:32:06.216978 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.216847    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dbhtr\" (UniqueName: \"kubernetes.io/projected/9bcdce67-2f01-4c5e-a20f-c976159f8683-kube-api-access-dbhtr\") pod \"dns-default-w8bsv\" (UID: \"9bcdce67-2f01-4c5e-a20f-c976159f8683\") " pod="openshift-dns/dns-default-w8bsv"
May 20 15:32:06.216978 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.216871    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert\") pod \"ingress-canary-szz5l\" (UID: \"fbac1f01-44d4-4042-a375-5a6a14f94d47\") " pod="openshift-ingress-canary/ingress-canary-szz5l"
May 20 15:32:06.216978 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.216901    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nwn4h\" (UniqueName: \"kubernetes.io/projected/fbac1f01-44d4-4042-a375-5a6a14f94d47-kube-api-access-nwn4h\") pod \"ingress-canary-szz5l\" (UID: \"fbac1f01-44d4-4042-a375-5a6a14f94d47\") " pod="openshift-ingress-canary/ingress-canary-szz5l"
May 20 15:32:06.217205 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:06.216983    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 15:32:06.217205 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:06.217096    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls podName:9bcdce67-2f01-4c5e-a20f-c976159f8683 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:06.717073561 +0000 UTC m=+35.655915110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls") pod "dns-default-w8bsv" (UID: "9bcdce67-2f01-4c5e-a20f-c976159f8683") : secret "dns-default-metrics-tls" not found
May 20 15:32:06.217319 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.217228    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bcdce67-2f01-4c5e-a20f-c976159f8683-tmp-dir\") pod \"dns-default-w8bsv\" (UID: \"9bcdce67-2f01-4c5e-a20f-c976159f8683\") " pod="openshift-dns/dns-default-w8bsv"
May 20 15:32:06.217369 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:06.217326    2568 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 15:32:06.217369 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.217345    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9bcdce67-2f01-4c5e-a20f-c976159f8683-config-volume\") pod \"dns-default-w8bsv\" (UID: \"9bcdce67-2f01-4c5e-a20f-c976159f8683\") " pod="openshift-dns/dns-default-w8bsv"
May 20 15:32:06.217455 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:06.217382    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert podName:fbac1f01-44d4-4042-a375-5a6a14f94d47 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:06.717365393 +0000 UTC m=+35.656206930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert") pod "ingress-canary-szz5l" (UID: "fbac1f01-44d4-4042-a375-5a6a14f94d47") : secret "canary-serving-cert" not found
May 20 15:32:06.228270 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.228243    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbhtr\" (UniqueName: \"kubernetes.io/projected/9bcdce67-2f01-4c5e-a20f-c976159f8683-kube-api-access-dbhtr\") pod \"dns-default-w8bsv\" (UID: \"9bcdce67-2f01-4c5e-a20f-c976159f8683\") " pod="openshift-dns/dns-default-w8bsv"
May 20 15:32:06.228418 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.228256    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwn4h\" (UniqueName: \"kubernetes.io/projected/fbac1f01-44d4-4042-a375-5a6a14f94d47-kube-api-access-nwn4h\") pod \"ingress-canary-szz5l\" (UID: \"fbac1f01-44d4-4042-a375-5a6a14f94d47\") " pod="openshift-ingress-canary/ingress-canary-szz5l"
May 20 15:32:06.620570 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.620524    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:06.621124 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.620670    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-2fdvj\" (UID: \"94373970-3fe5-4bbf-a05a-f4913b27e8d0\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj"
May 20 15:32:06.621124 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:06.620691    2568 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 15:32:06.621124 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:06.620708    2568 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666497c696-xxwnj: secret "image-registry-tls" not found
May 20 15:32:06.621124 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:06.620753    2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 20 15:32:06.621124 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:06.620784    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls podName:c010265c-2cd5-4e4f-9fd5-f75755dc1419 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:07.620765299 +0000 UTC m=+36.559606848 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls") pod "image-registry-666497c696-xxwnj" (UID: "c010265c-2cd5-4e4f-9fd5-f75755dc1419") : secret "image-registry-tls" not found
May 20 15:32:06.621124 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:06.620814    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert podName:94373970-3fe5-4bbf-a05a-f4913b27e8d0 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:07.62080412 +0000 UTC m=+36.559645655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-2fdvj" (UID: "94373970-3fe5-4bbf-a05a-f4913b27e8d0") : secret "networking-console-plugin-cert" not found
May 20 15:32:06.721681 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.721643    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls\") pod \"dns-default-w8bsv\" (UID: \"9bcdce67-2f01-4c5e-a20f-c976159f8683\") " pod="openshift-dns/dns-default-w8bsv"
May 20 15:32:06.721681 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:06.721679    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert\") pod \"ingress-canary-szz5l\" (UID: \"fbac1f01-44d4-4042-a375-5a6a14f94d47\") " pod="openshift-ingress-canary/ingress-canary-szz5l"
May 20 15:32:06.721876 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:06.721790    2568 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 15:32:06.721876 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:06.721796    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 15:32:06.721876 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:06.721844    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert podName:fbac1f01-44d4-4042-a375-5a6a14f94d47 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:07.721830062 +0000 UTC m=+36.660671597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert") pod "ingress-canary-szz5l" (UID: "fbac1f01-44d4-4042-a375-5a6a14f94d47") : secret "canary-serving-cert" not found
May 20 15:32:06.721876 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:06.721857    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls podName:9bcdce67-2f01-4c5e-a20f-c976159f8683 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:07.721851652 +0000 UTC m=+36.660693187 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls") pod "dns-default-w8bsv" (UID: "9bcdce67-2f01-4c5e-a20f-c976159f8683") : secret "dns-default-metrics-tls" not found
May 20 15:32:07.629197 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:07.629012    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-2fdvj\" (UID: \"94373970-3fe5-4bbf-a05a-f4913b27e8d0\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj"
May 20 15:32:07.629577 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:07.629225    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:07.629577 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:07.629155    2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 20 15:32:07.629577 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:07.629349    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert podName:94373970-3fe5-4bbf-a05a-f4913b27e8d0 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:09.629328306 +0000 UTC m=+38.568169843 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-2fdvj" (UID: "94373970-3fe5-4bbf-a05a-f4913b27e8d0") : secret "networking-console-plugin-cert" not found
May 20 15:32:07.629577 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:07.629356    2568 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 15:32:07.629577 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:07.629367    2568 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666497c696-xxwnj: secret "image-registry-tls" not found
May 20 15:32:07.629577 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:07.629398    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls podName:c010265c-2cd5-4e4f-9fd5-f75755dc1419 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:09.629387738 +0000 UTC m=+38.568229277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls") pod "image-registry-666497c696-xxwnj" (UID: "c010265c-2cd5-4e4f-9fd5-f75755dc1419") : secret "image-registry-tls" not found
May 20 15:32:07.642410 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:07.642346    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:32:07.642508 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:07.642432    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:32:07.642549 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:07.642531    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:32:07.645052 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:07.645011    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 20 15:32:07.645666 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:07.645650    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sz56g\""
May 20 15:32:07.645750 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:07.645671    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pq86x\""
May 20 15:32:07.645750 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:07.645680    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 20 15:32:07.645750 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:07.645688    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 20 15:32:07.645876 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:07.645652    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 20 15:32:07.730292 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:07.730258    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls\") pod \"dns-default-w8bsv\" (UID: \"9bcdce67-2f01-4c5e-a20f-c976159f8683\") " pod="openshift-dns/dns-default-w8bsv"
May 20 15:32:07.730292 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:07.730293    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert\") pod \"ingress-canary-szz5l\" (UID: \"fbac1f01-44d4-4042-a375-5a6a14f94d47\") " pod="openshift-ingress-canary/ingress-canary-szz5l"
May 20 15:32:07.730479 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:07.730403    2568 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 15:32:07.730479 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:07.730460    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 15:32:07.730479 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:07.730470    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert podName:fbac1f01-44d4-4042-a375-5a6a14f94d47 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:09.730451168 +0000 UTC m=+38.669292705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert") pod "ingress-canary-szz5l" (UID: "fbac1f01-44d4-4042-a375-5a6a14f94d47") : secret "canary-serving-cert" not found
May 20 15:32:07.730575 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:07.730506    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls podName:9bcdce67-2f01-4c5e-a20f-c976159f8683 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:09.730495077 +0000 UTC m=+38.669336613 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls") pod "dns-default-w8bsv" (UID: "9bcdce67-2f01-4c5e-a20f-c976159f8683") : secret "dns-default-metrics-tls" not found
May 20 15:32:07.876551 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:07.876515    2568 generic.go:358] "Generic (PLEG): container finished" podID="799a84c1-8372-4b99-a34d-105048106d5d" containerID="d93769f2a36bbb20bacccdd679ccc3141d3e191a7152028e160a9bdffd64f63a" exitCode=0
May 20 15:32:07.876707 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:07.876571    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvrb6" event={"ID":"799a84c1-8372-4b99-a34d-105048106d5d","Type":"ContainerDied","Data":"d93769f2a36bbb20bacccdd679ccc3141d3e191a7152028e160a9bdffd64f63a"}
May 20 15:32:08.882066 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:08.882014    2568 generic.go:358] "Generic (PLEG): container finished" podID="799a84c1-8372-4b99-a34d-105048106d5d" containerID="c67b03a9c2b693ea5258b222d163beaeea3a4734620d16068239b829c1dd47f7" exitCode=0
May 20 15:32:08.882542 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:08.882092    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvrb6" event={"ID":"799a84c1-8372-4b99-a34d-105048106d5d","Type":"ContainerDied","Data":"c67b03a9c2b693ea5258b222d163beaeea3a4734620d16068239b829c1dd47f7"}
May 20 15:32:09.647946 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:09.647913    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-2fdvj\" (UID: \"94373970-3fe5-4bbf-a05a-f4913b27e8d0\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj"
May 20 15:32:09.648128 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:09.647957    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:09.648128 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:09.648071    2568 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 15:32:09.648128 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:09.648083    2568 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666497c696-xxwnj: secret "image-registry-tls" not found
May 20 15:32:09.648128 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:09.648083    2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 20 15:32:09.648128 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:09.648131    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls podName:c010265c-2cd5-4e4f-9fd5-f75755dc1419 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:13.648116935 +0000 UTC m=+42.586958470 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls") pod "image-registry-666497c696-xxwnj" (UID: "c010265c-2cd5-4e4f-9fd5-f75755dc1419") : secret "image-registry-tls" not found
May 20 15:32:09.648289 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:09.648143    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert podName:94373970-3fe5-4bbf-a05a-f4913b27e8d0 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:13.648137507 +0000 UTC m=+42.586979041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-2fdvj" (UID: "94373970-3fe5-4bbf-a05a-f4913b27e8d0") : secret "networking-console-plugin-cert" not found
May 20 15:32:09.748903 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:09.748864    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls\") pod \"dns-default-w8bsv\" (UID: \"9bcdce67-2f01-4c5e-a20f-c976159f8683\") " pod="openshift-dns/dns-default-w8bsv"
May 20 15:32:09.748903 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:09.748903    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert\") pod \"ingress-canary-szz5l\" (UID: \"fbac1f01-44d4-4042-a375-5a6a14f94d47\") " pod="openshift-ingress-canary/ingress-canary-szz5l"
May 20 15:32:09.749125 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:09.749014    2568 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 15:32:09.749125 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:09.749034    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 15:32:09.749125 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:09.749082    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert podName:fbac1f01-44d4-4042-a375-5a6a14f94d47 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:13.749068485 +0000 UTC m=+42.687910021 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert") pod "ingress-canary-szz5l" (UID: "fbac1f01-44d4-4042-a375-5a6a14f94d47") : secret "canary-serving-cert" not found
May 20 15:32:09.749125 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:09.749096    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls podName:9bcdce67-2f01-4c5e-a20f-c976159f8683 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:13.749090177 +0000 UTC m=+42.687931712 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls") pod "dns-default-w8bsv" (UID: "9bcdce67-2f01-4c5e-a20f-c976159f8683") : secret "dns-default-metrics-tls" not found
May 20 15:32:09.887071 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:09.887031    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvrb6" event={"ID":"799a84c1-8372-4b99-a34d-105048106d5d","Type":"ContainerStarted","Data":"67210d08fed2bfb968256a7feb6db9115995dfd15c674e9a922858e6cc1f5092"}
May 20 15:32:09.909893 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:09.909776    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kvrb6" podStartSLOduration=5.84863806 podStartE2EDuration="38.909760642s" podCreationTimestamp="2026-05-20 15:31:31 +0000 UTC" firstStartedPulling="2026-05-20 15:31:34.110717669 +0000 UTC m=+3.049559204" lastFinishedPulling="2026-05-20 15:32:07.17184025 +0000 UTC m=+36.110681786" observedRunningTime="2026-05-20 15:32:09.908884194 +0000 UTC m=+38.847725776" watchObservedRunningTime="2026-05-20 15:32:09.909760642 +0000 UTC m=+38.848602199"
May 20 15:32:11.762733 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:11.762702    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb6d0d3d-79d3-4f3f-8f8e-974108118563-original-pull-secret\") pod \"global-pull-secret-syncer-cknlk\" (UID: \"cb6d0d3d-79d3-4f3f-8f8e-974108118563\") " pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:32:11.766048 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:11.766005    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb6d0d3d-79d3-4f3f-8f8e-974108118563-original-pull-secret\") pod \"global-pull-secret-syncer-cknlk\" (UID: \"cb6d0d3d-79d3-4f3f-8f8e-974108118563\") " pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:32:11.852133 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:11.852094    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cknlk"
May 20 15:32:12.068820 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:12.068789    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cknlk"]
May 20 15:32:12.072296 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:32:12.072269    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6d0d3d_79d3_4f3f_8f8e_974108118563.slice/crio-a35e4c620ac03b216e19b82a1e16bbc19eee8c308bee08c2a1dde1186d844b02 WatchSource:0}: Error finding container a35e4c620ac03b216e19b82a1e16bbc19eee8c308bee08c2a1dde1186d844b02: Status 404 returned error can't find the container with id a35e4c620ac03b216e19b82a1e16bbc19eee8c308bee08c2a1dde1186d844b02
May 20 15:32:12.896564 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:12.896521    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cknlk" event={"ID":"cb6d0d3d-79d3-4f3f-8f8e-974108118563","Type":"ContainerStarted","Data":"a35e4c620ac03b216e19b82a1e16bbc19eee8c308bee08c2a1dde1186d844b02"}
May 20 15:32:13.676996 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:13.676939    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-2fdvj\" (UID: \"94373970-3fe5-4bbf-a05a-f4913b27e8d0\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj"
May 20 15:32:13.677204 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:13.677047    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:13.677204 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:13.677147    2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 20 15:32:13.677313 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:13.677215    2568 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 15:32:13.677313 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:13.677229    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert podName:94373970-3fe5-4bbf-a05a-f4913b27e8d0 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:21.677213094 +0000 UTC m=+50.616054630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-2fdvj" (UID: "94373970-3fe5-4bbf-a05a-f4913b27e8d0") : secret "networking-console-plugin-cert" not found
May 20 15:32:13.677313 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:13.677231    2568 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666497c696-xxwnj: secret "image-registry-tls" not found
May 20 15:32:13.677313 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:13.677297    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls podName:c010265c-2cd5-4e4f-9fd5-f75755dc1419 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:21.677283581 +0000 UTC m=+50.616125115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls") pod "image-registry-666497c696-xxwnj" (UID: "c010265c-2cd5-4e4f-9fd5-f75755dc1419") : secret "image-registry-tls" not found
May 20 15:32:13.778361 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:13.778316    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls\") pod \"dns-default-w8bsv\" (UID: \"9bcdce67-2f01-4c5e-a20f-c976159f8683\") " pod="openshift-dns/dns-default-w8bsv"
May 20 15:32:13.778361 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:13.778366    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert\") pod \"ingress-canary-szz5l\" (UID: \"fbac1f01-44d4-4042-a375-5a6a14f94d47\") " pod="openshift-ingress-canary/ingress-canary-szz5l"
May 20 15:32:13.778613 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:13.778471    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 15:32:13.778613 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:13.778505    2568 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 15:32:13.778613 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:13.778544    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls podName:9bcdce67-2f01-4c5e-a20f-c976159f8683 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:21.77852503 +0000 UTC m=+50.717366566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls") pod "dns-default-w8bsv" (UID: "9bcdce67-2f01-4c5e-a20f-c976159f8683") : secret "dns-default-metrics-tls" not found
May 20 15:32:13.778613 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:13.778566    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert podName:fbac1f01-44d4-4042-a375-5a6a14f94d47 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:21.778551026 +0000 UTC m=+50.717392579 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert") pod "ingress-canary-szz5l" (UID: "fbac1f01-44d4-4042-a375-5a6a14f94d47") : secret "canary-serving-cert" not found
May 20 15:32:16.906622 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:16.906537    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cknlk" event={"ID":"cb6d0d3d-79d3-4f3f-8f8e-974108118563","Type":"ContainerStarted","Data":"267a2c1d1e556ac1ec068822d073be457548ca42ce39ef1d625efbf96414a09c"}
May 20 15:32:16.923209 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:16.923163    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cknlk" podStartSLOduration=33.383996388 podStartE2EDuration="37.923148667s" podCreationTimestamp="2026-05-20 15:31:39 +0000 UTC" firstStartedPulling="2026-05-20 15:32:12.074101097 +0000 UTC m=+41.012942633" lastFinishedPulling="2026-05-20 15:32:16.613253375 +0000 UTC m=+45.552094912" observedRunningTime="2026-05-20 15:32:16.922541352 +0000 UTC m=+45.861382910" watchObservedRunningTime="2026-05-20 15:32:16.923148667 +0000 UTC m=+45.861990273"
May 20 15:32:21.738953 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:21.738913    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-2fdvj\" (UID: \"94373970-3fe5-4bbf-a05a-f4913b27e8d0\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj"
May 20 15:32:21.738953 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:21.738958    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:21.739446 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:21.739061    2568 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 15:32:21.739446 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:21.739067    2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 20 15:32:21.739446 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:21.739131    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert podName:94373970-3fe5-4bbf-a05a-f4913b27e8d0 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:37.739115486 +0000 UTC m=+66.677957021 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-2fdvj" (UID: "94373970-3fe5-4bbf-a05a-f4913b27e8d0") : secret "networking-console-plugin-cert" not found
May 20 15:32:21.739446 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:21.739072    2568 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666497c696-xxwnj: secret "image-registry-tls" not found
May 20 15:32:21.739446 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:21.739221    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls podName:c010265c-2cd5-4e4f-9fd5-f75755dc1419 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:37.739202847 +0000 UTC m=+66.678044383 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls") pod "image-registry-666497c696-xxwnj" (UID: "c010265c-2cd5-4e4f-9fd5-f75755dc1419") : secret "image-registry-tls" not found
May 20 15:32:21.839631 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:21.839598    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls\") pod \"dns-default-w8bsv\" (UID: \"9bcdce67-2f01-4c5e-a20f-c976159f8683\") " pod="openshift-dns/dns-default-w8bsv"
May 20 15:32:21.839631 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:21.839631    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert\") pod \"ingress-canary-szz5l\" (UID: \"fbac1f01-44d4-4042-a375-5a6a14f94d47\") " pod="openshift-ingress-canary/ingress-canary-szz5l"
May 20 15:32:21.839831 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:21.839723    2568 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 15:32:21.839831 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:21.839730    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 15:32:21.839831 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:21.839775    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert podName:fbac1f01-44d4-4042-a375-5a6a14f94d47 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:37.839761691 +0000 UTC m=+66.778603226 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert") pod "ingress-canary-szz5l" (UID: "fbac1f01-44d4-4042-a375-5a6a14f94d47") : secret "canary-serving-cert" not found
May 20 15:32:21.839831 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:21.839787    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls podName:9bcdce67-2f01-4c5e-a20f-c976159f8683 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:37.839781731 +0000 UTC m=+66.778623266 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls") pod "dns-default-w8bsv" (UID: "9bcdce67-2f01-4c5e-a20f-c976159f8683") : secret "dns-default-metrics-tls" not found
May 20 15:32:33.876970 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:33.876944    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gqhp4"
May 20 15:32:37.448635 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:37.448599    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs\") pod \"network-metrics-daemon-skk8q\" (UID: \"3022a27a-6112-46ee-a54b-c5af5bee2a1f\") " pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:32:37.451418 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:37.451397    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 20 15:32:37.459408 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:37.459393    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 20 15:32:37.459471 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:37.459458    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs podName:3022a27a-6112-46ee-a54b-c5af5bee2a1f nodeName:}" failed. No retries permitted until 2026-05-20 15:33:41.459441542 +0000 UTC m=+130.398283077 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs") pod "network-metrics-daemon-skk8q" (UID: "3022a27a-6112-46ee-a54b-c5af5bee2a1f") : secret "metrics-daemon-secret" not found
May 20 15:32:37.549625 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:37.549598    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-slmsd\" (UniqueName: \"kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd\") pod \"network-check-target-r9x8t\" (UID: \"69c7e426-cc15-4efc-afbe-77377c1c3779\") " pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:32:37.552344 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:37.552330    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 20 15:32:37.562725 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:37.562711    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 20 15:32:37.573178 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:37.573158    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slmsd\" (UniqueName: \"kubernetes.io/projected/69c7e426-cc15-4efc-afbe-77377c1c3779-kube-api-access-slmsd\") pod \"network-check-target-r9x8t\" (UID: \"69c7e426-cc15-4efc-afbe-77377c1c3779\") " pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:32:37.660862 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:37.660837    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pq86x\""
May 20 15:32:37.668912 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:37.668898    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:32:37.750867 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:37.750825    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-2fdvj\" (UID: \"94373970-3fe5-4bbf-a05a-f4913b27e8d0\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj"
May 20 15:32:37.750999 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:37.750893    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:32:37.751068 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:37.751046    2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 20 15:32:37.751105 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:37.751084    2568 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 15:32:37.751105 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:37.751100    2568 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666497c696-xxwnj: secret "image-registry-tls" not found
May 20 15:32:37.751164 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:37.751122    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert podName:94373970-3fe5-4bbf-a05a-f4913b27e8d0 nodeName:}" failed. No retries permitted until 2026-05-20 15:33:09.751106716 +0000 UTC m=+98.689948251 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-2fdvj" (UID: "94373970-3fe5-4bbf-a05a-f4913b27e8d0") : secret "networking-console-plugin-cert" not found
May 20 15:32:37.751164 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:37.751162    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls podName:c010265c-2cd5-4e4f-9fd5-f75755dc1419 nodeName:}" failed. No retries permitted until 2026-05-20 15:33:09.75114541 +0000 UTC m=+98.689986947 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls") pod "image-registry-666497c696-xxwnj" (UID: "c010265c-2cd5-4e4f-9fd5-f75755dc1419") : secret "image-registry-tls" not found
May 20 15:32:37.806709 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:37.806676    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r9x8t"]
May 20 15:32:37.810878 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:32:37.810853    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c7e426_cc15_4efc_afbe_77377c1c3779.slice/crio-757fa3efc5ec92c928cb1b1de0a64bb557f29762dae814a83dbec8564c4bc502 WatchSource:0}: Error finding container 757fa3efc5ec92c928cb1b1de0a64bb557f29762dae814a83dbec8564c4bc502: Status 404 returned error can't find the container with id 757fa3efc5ec92c928cb1b1de0a64bb557f29762dae814a83dbec8564c4bc502
May 20 15:32:37.851869 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:37.851847    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls\") pod \"dns-default-w8bsv\" (UID: \"9bcdce67-2f01-4c5e-a20f-c976159f8683\") " pod="openshift-dns/dns-default-w8bsv"
May 20 15:32:37.851955 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:37.851877    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert\") pod \"ingress-canary-szz5l\" (UID: \"fbac1f01-44d4-4042-a375-5a6a14f94d47\") " pod="openshift-ingress-canary/ingress-canary-szz5l"
May 20 15:32:37.852012 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:37.851998    2568 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 15:32:37.852091 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:37.852080    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert podName:fbac1f01-44d4-4042-a375-5a6a14f94d47 nodeName:}" failed. No retries permitted until 2026-05-20 15:33:09.852067001 +0000 UTC m=+98.790908535 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert") pod "ingress-canary-szz5l" (UID: "fbac1f01-44d4-4042-a375-5a6a14f94d47") : secret "canary-serving-cert" not found
May 20 15:32:37.852132 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:37.851998    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 15:32:37.852132 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:32:37.852122    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls podName:9bcdce67-2f01-4c5e-a20f-c976159f8683 nodeName:}" failed. No retries permitted until 2026-05-20 15:33:09.852114078 +0000 UTC m=+98.790955613 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls") pod "dns-default-w8bsv" (UID: "9bcdce67-2f01-4c5e-a20f-c976159f8683") : secret "dns-default-metrics-tls" not found
May 20 15:32:37.944438 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:37.944412    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r9x8t" event={"ID":"69c7e426-cc15-4efc-afbe-77377c1c3779","Type":"ContainerStarted","Data":"757fa3efc5ec92c928cb1b1de0a64bb557f29762dae814a83dbec8564c4bc502"}
May 20 15:32:41.953988 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:41.953928    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r9x8t" event={"ID":"69c7e426-cc15-4efc-afbe-77377c1c3779","Type":"ContainerStarted","Data":"5972f96fb7ae224c55c9d3c4f1df10b44dbe5b4fbe560d2a8a9b0d55a75d60a4"}
May 20 15:32:41.954424 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:32:41.954069    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:33:09.799301 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:33:09.799134    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:33:09.799301 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:33:09.799251    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-2fdvj\" (UID: \"94373970-3fe5-4bbf-a05a-f4913b27e8d0\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj"
May 20 15:33:09.799301 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:33:09.799284    2568 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 15:33:09.799301 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:33:09.799309    2568 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666497c696-xxwnj: secret "image-registry-tls" not found
May 20 15:33:09.799850 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:33:09.799349    2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 20 15:33:09.799850 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:33:09.799379    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls podName:c010265c-2cd5-4e4f-9fd5-f75755dc1419 nodeName:}" failed. No retries permitted until 2026-05-20 15:34:13.799363906 +0000 UTC m=+162.738205442 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls") pod "image-registry-666497c696-xxwnj" (UID: "c010265c-2cd5-4e4f-9fd5-f75755dc1419") : secret "image-registry-tls" not found
May 20 15:33:09.799850 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:33:09.799412    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert podName:94373970-3fe5-4bbf-a05a-f4913b27e8d0 nodeName:}" failed. No retries permitted until 2026-05-20 15:34:13.799394917 +0000 UTC m=+162.738236465 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-2fdvj" (UID: "94373970-3fe5-4bbf-a05a-f4913b27e8d0") : secret "networking-console-plugin-cert" not found
May 20 15:33:09.900327 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:33:09.900290    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls\") pod \"dns-default-w8bsv\" (UID: \"9bcdce67-2f01-4c5e-a20f-c976159f8683\") " pod="openshift-dns/dns-default-w8bsv"
May 20 15:33:09.900327 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:33:09.900328    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert\") pod \"ingress-canary-szz5l\" (UID: \"fbac1f01-44d4-4042-a375-5a6a14f94d47\") " pod="openshift-ingress-canary/ingress-canary-szz5l"
May 20 15:33:09.900592 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:33:09.900451    2568 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 15:33:09.900592 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:33:09.900458    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 15:33:09.900592 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:33:09.900512    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert podName:fbac1f01-44d4-4042-a375-5a6a14f94d47 nodeName:}" failed. No retries permitted until 2026-05-20 15:34:13.900497985 +0000 UTC m=+162.839339520 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert") pod "ingress-canary-szz5l" (UID: "fbac1f01-44d4-4042-a375-5a6a14f94d47") : secret "canary-serving-cert" not found
May 20 15:33:09.900592 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:33:09.900525    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls podName:9bcdce67-2f01-4c5e-a20f-c976159f8683 nodeName:}" failed. No retries permitted until 2026-05-20 15:34:13.900518617 +0000 UTC m=+162.839360152 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls") pod "dns-default-w8bsv" (UID: "9bcdce67-2f01-4c5e-a20f-c976159f8683") : secret "dns-default-metrics-tls" not found
May 20 15:33:12.958103 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:33:12.958069    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-r9x8t"
May 20 15:33:12.975043 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:33:12.974979    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-r9x8t" podStartSLOduration=98.341068374 podStartE2EDuration="1m41.974965706s" podCreationTimestamp="2026-05-20 15:31:31 +0000 UTC" firstStartedPulling="2026-05-20 15:32:37.812283666 +0000 UTC m=+66.751125201" lastFinishedPulling="2026-05-20 15:32:41.446180999 +0000 UTC m=+70.385022533" observedRunningTime="2026-05-20 15:32:41.99049359 +0000 UTC m=+70.929335150" watchObservedRunningTime="2026-05-20 15:33:12.974965706 +0000 UTC m=+101.913807241"
May 20 15:33:41.533436 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:33:41.533398    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs\") pod \"network-metrics-daemon-skk8q\" (UID: \"3022a27a-6112-46ee-a54b-c5af5bee2a1f\") " pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:33:41.533857 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:33:41.533562    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 20 15:33:41.533857 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:33:41.533644    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs podName:3022a27a-6112-46ee-a54b-c5af5bee2a1f nodeName:}" failed. No retries permitted until 2026-05-20 15:35:43.533626742 +0000 UTC m=+252.472468276 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs") pod "network-metrics-daemon-skk8q" (UID: "3022a27a-6112-46ee-a54b-c5af5bee2a1f") : secret "metrics-daemon-secret" not found
May 20 15:34:03.431247 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.431217    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6"]
May 20 15:34:03.433037 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.432991    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6"
May 20 15:34:03.433910 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.433889    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-8ltwr"]
May 20 15:34:03.435621 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.435606    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-8ltwr"
May 20 15:34:03.436385 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.436360    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 20 15:34:03.436451 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.436435    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 20 15:34:03.438120 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.438082    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
May 20 15:34:03.438289 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.438150    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ctbkt\""
May 20 15:34:03.438289 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.438161    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
May 20 15:34:03.438409 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.438396    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 20 15:34:03.438733 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.438715    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 20 15:34:03.438823 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.438731    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
May 20 15:34:03.438823 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.438811    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
May 20 15:34:03.438969 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.438952    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ptnmb\""
May 20 15:34:03.444804 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.444784    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
May 20 15:34:03.446378 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.446345    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6"]
May 20 15:34:03.451839 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.451817    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-8ltwr"]
May 20 15:34:03.494470 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.494433    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cbb5691-4ce1-46dd-8a53-d94b5c19edcf-tmp\") pod \"insights-operator-78bdfb888c-8ltwr\" (UID: \"0cbb5691-4ce1-46dd-8a53-d94b5c19edcf\") " pod="openshift-insights/insights-operator-78bdfb888c-8ltwr"
May 20 15:34:03.494470 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.494468    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cbb5691-4ce1-46dd-8a53-d94b5c19edcf-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-8ltwr\" (UID: \"0cbb5691-4ce1-46dd-8a53-d94b5c19edcf\") " pod="openshift-insights/insights-operator-78bdfb888c-8ltwr"
May 20 15:34:03.494722 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.494484    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cbb5691-4ce1-46dd-8a53-d94b5c19edcf-service-ca-bundle\") pod \"insights-operator-78bdfb888c-8ltwr\" (UID: \"0cbb5691-4ce1-46dd-8a53-d94b5c19edcf\") " pod="openshift-insights/insights-operator-78bdfb888c-8ltwr"
May 20 15:34:03.494722 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.494501    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45gvj\" (UniqueName: \"kubernetes.io/projected/0cbb5691-4ce1-46dd-8a53-d94b5c19edcf-kube-api-access-45gvj\") pod \"insights-operator-78bdfb888c-8ltwr\" (UID: \"0cbb5691-4ce1-46dd-8a53-d94b5c19edcf\") " pod="openshift-insights/insights-operator-78bdfb888c-8ltwr"
May 20 15:34:03.494722 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.494630    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2vrc\" (UniqueName: \"kubernetes.io/projected/e984d532-c6fd-4d79-8b16-07fc216f278d-kube-api-access-k2vrc\") pod \"cluster-monitoring-operator-6c75959455-c85d6\" (UID: \"e984d532-c6fd-4d79-8b16-07fc216f278d\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6"
May 20 15:34:03.494722 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.494668    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0cbb5691-4ce1-46dd-8a53-d94b5c19edcf-snapshots\") pod \"insights-operator-78bdfb888c-8ltwr\" (UID: \"0cbb5691-4ce1-46dd-8a53-d94b5c19edcf\") " pod="openshift-insights/insights-operator-78bdfb888c-8ltwr"
May 20 15:34:03.494722 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.494719    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0cbb5691-4ce1-46dd-8a53-d94b5c19edcf-serving-cert\") pod \"insights-operator-78bdfb888c-8ltwr\" (UID: \"0cbb5691-4ce1-46dd-8a53-d94b5c19edcf\") " pod="openshift-insights/insights-operator-78bdfb888c-8ltwr"
May 20 15:34:03.494939 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.494766    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e984d532-c6fd-4d79-8b16-07fc216f278d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-c85d6\" (UID: \"e984d532-c6fd-4d79-8b16-07fc216f278d\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6"
May 20 15:34:03.494939 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.494817    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e984d532-c6fd-4d79-8b16-07fc216f278d-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-c85d6\" (UID: \"e984d532-c6fd-4d79-8b16-07fc216f278d\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6"
May 20 15:34:03.535501 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.535467    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-lz659"]
May 20 15:34:03.537614 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.537599    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-lz659"
May 20 15:34:03.539771 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.539747    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
May 20 15:34:03.539885 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.539848    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-pw2s4\""
May 20 15:34:03.539948 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.539899    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
May 20 15:34:03.547359 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.547338    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-lz659"]
May 20 15:34:03.595723 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.595693    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e984d532-c6fd-4d79-8b16-07fc216f278d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-c85d6\" (UID: \"e984d532-c6fd-4d79-8b16-07fc216f278d\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6"
May 20 15:34:03.595884 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.595735    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e984d532-c6fd-4d79-8b16-07fc216f278d-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-c85d6\" (UID: \"e984d532-c6fd-4d79-8b16-07fc216f278d\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6"
May 20 15:34:03.595884 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.595769    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cbb5691-4ce1-46dd-8a53-d94b5c19edcf-tmp\") pod \"insights-operator-78bdfb888c-8ltwr\" (UID: \"0cbb5691-4ce1-46dd-8a53-d94b5c19edcf\") " pod="openshift-insights/insights-operator-78bdfb888c-8ltwr"
May 20 15:34:03.595884 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.595785    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cbb5691-4ce1-46dd-8a53-d94b5c19edcf-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-8ltwr\" (UID: \"0cbb5691-4ce1-46dd-8a53-d94b5c19edcf\") " pod="openshift-insights/insights-operator-78bdfb888c-8ltwr"
May 20 15:34:03.595884 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.595802    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cbb5691-4ce1-46dd-8a53-d94b5c19edcf-service-ca-bundle\") pod \"insights-operator-78bdfb888c-8ltwr\" (UID: \"0cbb5691-4ce1-46dd-8a53-d94b5c19edcf\") " pod="openshift-insights/insights-operator-78bdfb888c-8ltwr"
May 20 15:34:03.595884 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.595818    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-45gvj\" (UniqueName: \"kubernetes.io/projected/0cbb5691-4ce1-46dd-8a53-d94b5c19edcf-kube-api-access-45gvj\") pod \"insights-operator-78bdfb888c-8ltwr\" (UID: \"0cbb5691-4ce1-46dd-8a53-d94b5c19edcf\") " pod="openshift-insights/insights-operator-78bdfb888c-8ltwr"
May 20 15:34:03.595884 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.595849    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k2vrc\" (UniqueName: \"kubernetes.io/projected/e984d532-c6fd-4d79-8b16-07fc216f278d-kube-api-access-k2vrc\") pod \"cluster-monitoring-operator-6c75959455-c85d6\" (UID: \"e984d532-c6fd-4d79-8b16-07fc216f278d\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6"
May 20 15:34:03.595884 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.595866    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0cbb5691-4ce1-46dd-8a53-d94b5c19edcf-snapshots\") pod \"insights-operator-78bdfb888c-8ltwr\" (UID: \"0cbb5691-4ce1-46dd-8a53-d94b5c19edcf\") " pod="openshift-insights/insights-operator-78bdfb888c-8ltwr"
May 20 15:34:03.596228 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.595903    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0cbb5691-4ce1-46dd-8a53-d94b5c19edcf-serving-cert\") pod \"insights-operator-78bdfb888c-8ltwr\" (UID: \"0cbb5691-4ce1-46dd-8a53-d94b5c19edcf\") " pod="openshift-insights/insights-operator-78bdfb888c-8ltwr"
May 20 15:34:03.596228 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.595936    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whktr\" (UniqueName: \"kubernetes.io/projected/3390fc6f-b1ea-40b1-b920-87f3f2f96a12-kube-api-access-whktr\") pod \"volume-data-source-validator-5cf8c89d64-lz659\" (UID: \"3390fc6f-b1ea-40b1-b920-87f3f2f96a12\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-lz659"
May 20 15:34:03.596228 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:03.595847    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 20 15:34:03.596228 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:03.596149    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e984d532-c6fd-4d79-8b16-07fc216f278d-cluster-monitoring-operator-tls podName:e984d532-c6fd-4d79-8b16-07fc216f278d nodeName:}" failed. No retries permitted until 2026-05-20 15:34:04.096127035 +0000 UTC m=+153.034968572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e984d532-c6fd-4d79-8b16-07fc216f278d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-c85d6" (UID: "e984d532-c6fd-4d79-8b16-07fc216f278d") : secret "cluster-monitoring-operator-tls" not found
May 20 15:34:03.596532 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.596510    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0cbb5691-4ce1-46dd-8a53-d94b5c19edcf-snapshots\") pod \"insights-operator-78bdfb888c-8ltwr\" (UID: \"0cbb5691-4ce1-46dd-8a53-d94b5c19edcf\") " pod="openshift-insights/insights-operator-78bdfb888c-8ltwr"
May 20 15:34:03.596602 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.596512    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cbb5691-4ce1-46dd-8a53-d94b5c19edcf-service-ca-bundle\") pod \"insights-operator-78bdfb888c-8ltwr\" (UID: \"0cbb5691-4ce1-46dd-8a53-d94b5c19edcf\") " pod="openshift-insights/insights-operator-78bdfb888c-8ltwr"
May 20 15:34:03.596643 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.596625    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e984d532-c6fd-4d79-8b16-07fc216f278d-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-c85d6\" (UID: \"e984d532-c6fd-4d79-8b16-07fc216f278d\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6"
May 20 15:34:03.596770 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.596752    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cbb5691-4ce1-46dd-8a53-d94b5c19edcf-tmp\") pod \"insights-operator-78bdfb888c-8ltwr\" (UID: \"0cbb5691-4ce1-46dd-8a53-d94b5c19edcf\") " pod="openshift-insights/insights-operator-78bdfb888c-8ltwr"
May 20 15:34:03.596808 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.596795    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cbb5691-4ce1-46dd-8a53-d94b5c19edcf-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-8ltwr\" (UID: \"0cbb5691-4ce1-46dd-8a53-d94b5c19edcf\") " pod="openshift-insights/insights-operator-78bdfb888c-8ltwr"
May 20 15:34:03.598378 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.598363    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0cbb5691-4ce1-46dd-8a53-d94b5c19edcf-serving-cert\") pod \"insights-operator-78bdfb888c-8ltwr\" (UID: \"0cbb5691-4ce1-46dd-8a53-d94b5c19edcf\") " pod="openshift-insights/insights-operator-78bdfb888c-8ltwr"
May 20 15:34:03.607998 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.607973    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45gvj\" (UniqueName: \"kubernetes.io/projected/0cbb5691-4ce1-46dd-8a53-d94b5c19edcf-kube-api-access-45gvj\") pod \"insights-operator-78bdfb888c-8ltwr\" (UID: \"0cbb5691-4ce1-46dd-8a53-d94b5c19edcf\") " pod="openshift-insights/insights-operator-78bdfb888c-8ltwr"
May 20 15:34:03.608157 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.608117    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2vrc\" (UniqueName: \"kubernetes.io/projected/e984d532-c6fd-4d79-8b16-07fc216f278d-kube-api-access-k2vrc\") pod \"cluster-monitoring-operator-6c75959455-c85d6\" (UID: \"e984d532-c6fd-4d79-8b16-07fc216f278d\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6"
May 20 15:34:03.638504 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.638472    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-x844v"]
May 20 15:34:03.640975 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.640939    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x844v"
May 20 15:34:03.643169 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.643144    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
May 20 15:34:03.643421 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.643398    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
May 20 15:34:03.643533 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.643438    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
May 20 15:34:03.643597 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.643563    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-jv25b\""
May 20 15:34:03.653238 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.653216    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-x844v"]
May 20 15:34:03.696652 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.696569    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-whktr\" (UniqueName: \"kubernetes.io/projected/3390fc6f-b1ea-40b1-b920-87f3f2f96a12-kube-api-access-whktr\") pod \"volume-data-source-validator-5cf8c89d64-lz659\" (UID: \"3390fc6f-b1ea-40b1-b920-87f3f2f96a12\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-lz659"
May 20 15:34:03.696789 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.696673    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5m78\" (UniqueName: \"kubernetes.io/projected/0dba48cd-33a2-4fde-a534-38579b96efe0-kube-api-access-p5m78\") pod \"cluster-samples-operator-697587599-x844v\" (UID: \"0dba48cd-33a2-4fde-a534-38579b96efe0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x844v"
May 20 15:34:03.696789 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.696747    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dba48cd-33a2-4fde-a534-38579b96efe0-samples-operator-tls\") pod \"cluster-samples-operator-697587599-x844v\" (UID: \"0dba48cd-33a2-4fde-a534-38579b96efe0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x844v"
May 20 15:34:03.710727 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.710695    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whktr\" (UniqueName: \"kubernetes.io/projected/3390fc6f-b1ea-40b1-b920-87f3f2f96a12-kube-api-access-whktr\") pod \"volume-data-source-validator-5cf8c89d64-lz659\" (UID: \"3390fc6f-b1ea-40b1-b920-87f3f2f96a12\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-lz659"
May 20 15:34:03.749152 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.749107    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-8ltwr"
May 20 15:34:03.797555 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.797523    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dba48cd-33a2-4fde-a534-38579b96efe0-samples-operator-tls\") pod \"cluster-samples-operator-697587599-x844v\" (UID: \"0dba48cd-33a2-4fde-a534-38579b96efe0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x844v"
May 20 15:34:03.797684 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.797657    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p5m78\" (UniqueName: \"kubernetes.io/projected/0dba48cd-33a2-4fde-a534-38579b96efe0-kube-api-access-p5m78\") pod \"cluster-samples-operator-697587599-x844v\" (UID: \"0dba48cd-33a2-4fde-a534-38579b96efe0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x844v"
May 20 15:34:03.797723 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:03.797701    2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 20 15:34:03.797798 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:03.797783    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dba48cd-33a2-4fde-a534-38579b96efe0-samples-operator-tls podName:0dba48cd-33a2-4fde-a534-38579b96efe0 nodeName:}" failed. No retries permitted until 2026-05-20 15:34:04.297760624 +0000 UTC m=+153.236602161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0dba48cd-33a2-4fde-a534-38579b96efe0-samples-operator-tls") pod "cluster-samples-operator-697587599-x844v" (UID: "0dba48cd-33a2-4fde-a534-38579b96efe0") : secret "samples-operator-tls" not found
May 20 15:34:03.808447 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.808413    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5m78\" (UniqueName: \"kubernetes.io/projected/0dba48cd-33a2-4fde-a534-38579b96efe0-kube-api-access-p5m78\") pod \"cluster-samples-operator-697587599-x844v\" (UID: \"0dba48cd-33a2-4fde-a534-38579b96efe0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x844v"
May 20 15:34:03.846477 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.846443    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-lz659"
May 20 15:34:03.875616 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.875591    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-8ltwr"]
May 20 15:34:03.878224 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:34:03.878185    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbb5691_4ce1_46dd_8a53_d94b5c19edcf.slice/crio-ecb5e7f598fd3777202758b893ae3a4a4662d43d111af90cf603f71bea56f9be WatchSource:0}: Error finding container ecb5e7f598fd3777202758b893ae3a4a4662d43d111af90cf603f71bea56f9be: Status 404 returned error can't find the container with id ecb5e7f598fd3777202758b893ae3a4a4662d43d111af90cf603f71bea56f9be
May 20 15:34:03.968124 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:03.968048    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-lz659"]
May 20 15:34:03.972099 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:34:03.972070    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3390fc6f_b1ea_40b1_b920_87f3f2f96a12.slice/crio-38f2f64421335690e841127cc8a695c8d960ffb3042b046f623dbd04500c0bea WatchSource:0}: Error finding container 38f2f64421335690e841127cc8a695c8d960ffb3042b046f623dbd04500c0bea: Status 404 returned error can't find the container with id 38f2f64421335690e841127cc8a695c8d960ffb3042b046f623dbd04500c0bea
May 20 15:34:04.100402 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.100359    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e984d532-c6fd-4d79-8b16-07fc216f278d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-c85d6\" (UID: \"e984d532-c6fd-4d79-8b16-07fc216f278d\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6"
May 20 15:34:04.100591 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:04.100508    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 20 15:34:04.100591 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:04.100574    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e984d532-c6fd-4d79-8b16-07fc216f278d-cluster-monitoring-operator-tls podName:e984d532-c6fd-4d79-8b16-07fc216f278d nodeName:}" failed. No retries permitted until 2026-05-20 15:34:05.100559717 +0000 UTC m=+154.039401255 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e984d532-c6fd-4d79-8b16-07fc216f278d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-c85d6" (UID: "e984d532-c6fd-4d79-8b16-07fc216f278d") : secret "cluster-monitoring-operator-tls" not found
May 20 15:34:04.106593 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.106565    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-8ltwr" event={"ID":"0cbb5691-4ce1-46dd-8a53-d94b5c19edcf","Type":"ContainerStarted","Data":"ecb5e7f598fd3777202758b893ae3a4a4662d43d111af90cf603f71bea56f9be"}
May 20 15:34:04.107502 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.107479    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-lz659" event={"ID":"3390fc6f-b1ea-40b1-b920-87f3f2f96a12","Type":"ContainerStarted","Data":"38f2f64421335690e841127cc8a695c8d960ffb3042b046f623dbd04500c0bea"}
May 20 15:34:04.301495 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.301406    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dba48cd-33a2-4fde-a534-38579b96efe0-samples-operator-tls\") pod \"cluster-samples-operator-697587599-x844v\" (UID: \"0dba48cd-33a2-4fde-a534-38579b96efe0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x844v"
May 20 15:34:04.301645 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:04.301552    2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 20 15:34:04.301645 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:04.301611    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dba48cd-33a2-4fde-a534-38579b96efe0-samples-operator-tls podName:0dba48cd-33a2-4fde-a534-38579b96efe0 nodeName:}" failed. No retries permitted until 2026-05-20 15:34:05.301597606 +0000 UTC m=+154.240439141 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0dba48cd-33a2-4fde-a534-38579b96efe0-samples-operator-tls") pod "cluster-samples-operator-697587599-x844v" (UID: "0dba48cd-33a2-4fde-a534-38579b96efe0") : secret "samples-operator-tls" not found
May 20 15:34:04.457786 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.457753    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mbpgr"]
May 20 15:34:04.459564 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.459549    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mbpgr"
May 20 15:34:04.461872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.461849    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
May 20 15:34:04.461994 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.461859    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-n999b\""
May 20 15:34:04.462173 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.462135    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
May 20 15:34:04.462285 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.462270    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
May 20 15:34:04.462657 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.462636    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
May 20 15:34:04.469431 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.469409    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mbpgr"]
May 20 15:34:04.503599 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.503568    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cktkb\" (UniqueName: \"kubernetes.io/projected/1bd01bf5-75e6-45d5-9bf3-0b480b5457e8-kube-api-access-cktkb\") pod \"kube-storage-version-migrator-operator-565644f49-mbpgr\" (UID: \"1bd01bf5-75e6-45d5-9bf3-0b480b5457e8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mbpgr"
May 20 15:34:04.503787 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.503630    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1bd01bf5-75e6-45d5-9bf3-0b480b5457e8-config\") pod \"kube-storage-version-migrator-operator-565644f49-mbpgr\" (UID: \"1bd01bf5-75e6-45d5-9bf3-0b480b5457e8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mbpgr"
May 20 15:34:04.503787 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.503761    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1bd01bf5-75e6-45d5-9bf3-0b480b5457e8-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-mbpgr\" (UID: \"1bd01bf5-75e6-45d5-9bf3-0b480b5457e8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mbpgr"
May 20 15:34:04.604910 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.604871    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cktkb\" (UniqueName: \"kubernetes.io/projected/1bd01bf5-75e6-45d5-9bf3-0b480b5457e8-kube-api-access-cktkb\") pod \"kube-storage-version-migrator-operator-565644f49-mbpgr\" (UID: \"1bd01bf5-75e6-45d5-9bf3-0b480b5457e8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mbpgr"
May 20 15:34:04.605107 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.604978    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1bd01bf5-75e6-45d5-9bf3-0b480b5457e8-config\") pod \"kube-storage-version-migrator-operator-565644f49-mbpgr\" (UID: \"1bd01bf5-75e6-45d5-9bf3-0b480b5457e8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mbpgr"
May 20 15:34:04.605107 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.605073    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1bd01bf5-75e6-45d5-9bf3-0b480b5457e8-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-mbpgr\" (UID: \"1bd01bf5-75e6-45d5-9bf3-0b480b5457e8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mbpgr"
May 20 15:34:04.605835 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.605802    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1bd01bf5-75e6-45d5-9bf3-0b480b5457e8-config\") pod \"kube-storage-version-migrator-operator-565644f49-mbpgr\" (UID: \"1bd01bf5-75e6-45d5-9bf3-0b480b5457e8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mbpgr"
May 20 15:34:04.608271 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.608245    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1bd01bf5-75e6-45d5-9bf3-0b480b5457e8-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-mbpgr\" (UID: \"1bd01bf5-75e6-45d5-9bf3-0b480b5457e8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mbpgr"
May 20 15:34:04.614430 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.614376    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cktkb\" (UniqueName: \"kubernetes.io/projected/1bd01bf5-75e6-45d5-9bf3-0b480b5457e8-kube-api-access-cktkb\") pod \"kube-storage-version-migrator-operator-565644f49-mbpgr\" (UID: \"1bd01bf5-75e6-45d5-9bf3-0b480b5457e8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mbpgr"
May 20 15:34:04.768848 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.768801    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mbpgr"
May 20 15:34:04.922826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:04.922798    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mbpgr"]
May 20 15:34:04.928747 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:34:04.928705    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd01bf5_75e6_45d5_9bf3_0b480b5457e8.slice/crio-8b64265b17ee52e244fcb706c5d6ac5c557ac6aca29f4a5517285cd22d58bad6 WatchSource:0}: Error finding container 8b64265b17ee52e244fcb706c5d6ac5c557ac6aca29f4a5517285cd22d58bad6: Status 404 returned error can't find the container with id 8b64265b17ee52e244fcb706c5d6ac5c557ac6aca29f4a5517285cd22d58bad6
May 20 15:34:05.108858 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:05.108826    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e984d532-c6fd-4d79-8b16-07fc216f278d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-c85d6\" (UID: \"e984d532-c6fd-4d79-8b16-07fc216f278d\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6"
May 20 15:34:05.109155 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:05.109080    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 20 15:34:05.109155 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:05.109144    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e984d532-c6fd-4d79-8b16-07fc216f278d-cluster-monitoring-operator-tls podName:e984d532-c6fd-4d79-8b16-07fc216f278d nodeName:}" failed. No retries permitted until 2026-05-20 15:34:07.109124354 +0000 UTC m=+156.047965890 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e984d532-c6fd-4d79-8b16-07fc216f278d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-c85d6" (UID: "e984d532-c6fd-4d79-8b16-07fc216f278d") : secret "cluster-monitoring-operator-tls" not found
May 20 15:34:05.111340 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:05.111310    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mbpgr" event={"ID":"1bd01bf5-75e6-45d5-9bf3-0b480b5457e8","Type":"ContainerStarted","Data":"8b64265b17ee52e244fcb706c5d6ac5c557ac6aca29f4a5517285cd22d58bad6"}
May 20 15:34:05.311086 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:05.311058    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dba48cd-33a2-4fde-a534-38579b96efe0-samples-operator-tls\") pod \"cluster-samples-operator-697587599-x844v\" (UID: \"0dba48cd-33a2-4fde-a534-38579b96efe0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x844v"
May 20 15:34:05.311238 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:05.311207    2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 20 15:34:05.311297 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:05.311264    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dba48cd-33a2-4fde-a534-38579b96efe0-samples-operator-tls podName:0dba48cd-33a2-4fde-a534-38579b96efe0 nodeName:}" failed. No retries permitted until 2026-05-20 15:34:07.311247853 +0000 UTC m=+156.250089389 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0dba48cd-33a2-4fde-a534-38579b96efe0-samples-operator-tls") pod "cluster-samples-operator-697587599-x844v" (UID: "0dba48cd-33a2-4fde-a534-38579b96efe0") : secret "samples-operator-tls" not found
May 20 15:34:06.115370 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:06.115314    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-8ltwr" event={"ID":"0cbb5691-4ce1-46dd-8a53-d94b5c19edcf","Type":"ContainerStarted","Data":"2da0fd8cf93192cd243883f2fc3e1c2ef041ce83bb910ec6383d76b6da43ae66"}
May 20 15:34:06.117042 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:06.116994    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-lz659" event={"ID":"3390fc6f-b1ea-40b1-b920-87f3f2f96a12","Type":"ContainerStarted","Data":"c8d51374ba0414cf58ab5dd4c16232f822fd9b2836f667406f68a9e3b7045169"}
May 20 15:34:06.132963 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:06.132914    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-operator-78bdfb888c-8ltwr" podStartSLOduration=1.091962778 podStartE2EDuration="3.132896938s" podCreationTimestamp="2026-05-20 15:34:03 +0000 UTC" firstStartedPulling="2026-05-20 15:34:03.879979175 +0000 UTC m=+152.818820710" lastFinishedPulling="2026-05-20 15:34:05.920913321 +0000 UTC m=+154.859754870" observedRunningTime="2026-05-20 15:34:06.132317636 +0000 UTC m=+155.071159193" watchObservedRunningTime="2026-05-20 15:34:06.132896938 +0000 UTC m=+155.071738494"
May 20 15:34:06.147682 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:06.147621    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-lz659" podStartSLOduration=1.820534731 podStartE2EDuration="3.14759934s" podCreationTimestamp="2026-05-20 15:34:03 +0000 UTC" firstStartedPulling="2026-05-20 15:34:03.974004819 +0000 UTC m=+152.912846354" lastFinishedPulling="2026-05-20 15:34:05.301069413 +0000 UTC m=+154.239910963" observedRunningTime="2026-05-20 15:34:06.146331247 +0000 UTC m=+155.085172805" watchObservedRunningTime="2026-05-20 15:34:06.14759934 +0000 UTC m=+155.086440900"
May 20 15:34:07.121764 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.121726    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mbpgr" event={"ID":"1bd01bf5-75e6-45d5-9bf3-0b480b5457e8","Type":"ContainerStarted","Data":"8656468435c128b20865d3316295fc289ca691693a1a231ed4b22fe11f477fb0"}
May 20 15:34:07.125660 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.125634    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e984d532-c6fd-4d79-8b16-07fc216f278d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-c85d6\" (UID: \"e984d532-c6fd-4d79-8b16-07fc216f278d\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6"
May 20 15:34:07.125783 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:07.125765    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 20 15:34:07.125858 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:07.125841    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e984d532-c6fd-4d79-8b16-07fc216f278d-cluster-monitoring-operator-tls podName:e984d532-c6fd-4d79-8b16-07fc216f278d nodeName:}" failed. No retries permitted until 2026-05-20 15:34:11.125820778 +0000 UTC m=+160.064662313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e984d532-c6fd-4d79-8b16-07fc216f278d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-c85d6" (UID: "e984d532-c6fd-4d79-8b16-07fc216f278d") : secret "cluster-monitoring-operator-tls" not found
May 20 15:34:07.138537 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.138487    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mbpgr" podStartSLOduration=1.141215695 podStartE2EDuration="3.138473743s" podCreationTimestamp="2026-05-20 15:34:04 +0000 UTC" firstStartedPulling="2026-05-20 15:34:04.93067261 +0000 UTC m=+153.869514150" lastFinishedPulling="2026-05-20 15:34:06.927930649 +0000 UTC m=+155.866772198" observedRunningTime="2026-05-20 15:34:07.137529193 +0000 UTC m=+156.076370750" watchObservedRunningTime="2026-05-20 15:34:07.138473743 +0000 UTC m=+156.077315299"
May 20 15:34:07.326947 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.326904    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dba48cd-33a2-4fde-a534-38579b96efe0-samples-operator-tls\") pod \"cluster-samples-operator-697587599-x844v\" (UID: \"0dba48cd-33a2-4fde-a534-38579b96efe0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x844v"
May 20 15:34:07.327221 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:07.327082    2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 20 15:34:07.327221 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:07.327158    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dba48cd-33a2-4fde-a534-38579b96efe0-samples-operator-tls podName:0dba48cd-33a2-4fde-a534-38579b96efe0 nodeName:}" failed. No retries permitted until 2026-05-20 15:34:11.327136044 +0000 UTC m=+160.265977582 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0dba48cd-33a2-4fde-a534-38579b96efe0-samples-operator-tls") pod "cluster-samples-operator-697587599-x844v" (UID: "0dba48cd-33a2-4fde-a534-38579b96efe0") : secret "samples-operator-tls" not found
May 20 15:34:07.328425 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.328398    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-7crcg"]
May 20 15:34:07.330597 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.330582    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-7crcg"
May 20 15:34:07.333577 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.333560    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-p267h\""
May 20 15:34:07.342778 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.342754    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-7crcg"]
May 20 15:34:07.428109 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.428064    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p29fj\" (UniqueName: \"kubernetes.io/projected/74f685e9-2b4e-47a2-b2e0-dd6158968cb3-kube-api-access-p29fj\") pod \"network-check-source-7f44b6dd87-7crcg\" (UID: \"74f685e9-2b4e-47a2-b2e0-dd6158968cb3\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-7crcg"
May 20 15:34:07.521548 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.521519    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-smrgr"]
May 20 15:34:07.523663 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.523634    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-smrgr"
May 20 15:34:07.526117 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.525840    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
May 20 15:34:07.526117 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.525879    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
May 20 15:34:07.526308 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.526154    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
May 20 15:34:07.526308 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.526247    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-hcx9n\""
May 20 15:34:07.526308 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.526249    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
May 20 15:34:07.528502 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.528479    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p29fj\" (UniqueName: \"kubernetes.io/projected/74f685e9-2b4e-47a2-b2e0-dd6158968cb3-kube-api-access-p29fj\") pod \"network-check-source-7f44b6dd87-7crcg\" (UID: \"74f685e9-2b4e-47a2-b2e0-dd6158968cb3\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-7crcg"
May 20 15:34:07.535332 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.535287    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-smrgr"]
May 20 15:34:07.538386 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.538360    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p29fj\" (UniqueName: \"kubernetes.io/projected/74f685e9-2b4e-47a2-b2e0-dd6158968cb3-kube-api-access-p29fj\") pod \"network-check-source-7f44b6dd87-7crcg\" (UID: \"74f685e9-2b4e-47a2-b2e0-dd6158968cb3\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-7crcg"
May 20 15:34:07.629977 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.629891    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq8fs\" (UniqueName: \"kubernetes.io/projected/afbc552d-4335-4a1c-a5e9-df2444a56e1d-kube-api-access-gq8fs\") pod \"service-ca-operator-57b5b68b8-smrgr\" (UID: \"afbc552d-4335-4a1c-a5e9-df2444a56e1d\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-smrgr"
May 20 15:34:07.630150 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.629980    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/afbc552d-4335-4a1c-a5e9-df2444a56e1d-serving-cert\") pod \"service-ca-operator-57b5b68b8-smrgr\" (UID: \"afbc552d-4335-4a1c-a5e9-df2444a56e1d\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-smrgr"
May 20 15:34:07.630150 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.630051    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/afbc552d-4335-4a1c-a5e9-df2444a56e1d-config\") pod \"service-ca-operator-57b5b68b8-smrgr\" (UID: \"afbc552d-4335-4a1c-a5e9-df2444a56e1d\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-smrgr"
May 20 15:34:07.638822 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.638795    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-7crcg"
May 20 15:34:07.730500 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.730467    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gq8fs\" (UniqueName: \"kubernetes.io/projected/afbc552d-4335-4a1c-a5e9-df2444a56e1d-kube-api-access-gq8fs\") pod \"service-ca-operator-57b5b68b8-smrgr\" (UID: \"afbc552d-4335-4a1c-a5e9-df2444a56e1d\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-smrgr"
May 20 15:34:07.730662 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.730533    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/afbc552d-4335-4a1c-a5e9-df2444a56e1d-serving-cert\") pod \"service-ca-operator-57b5b68b8-smrgr\" (UID: \"afbc552d-4335-4a1c-a5e9-df2444a56e1d\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-smrgr"
May 20 15:34:07.730748 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.730728    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/afbc552d-4335-4a1c-a5e9-df2444a56e1d-config\") pod \"service-ca-operator-57b5b68b8-smrgr\" (UID: \"afbc552d-4335-4a1c-a5e9-df2444a56e1d\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-smrgr"
May 20 15:34:07.731369 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.731345    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/afbc552d-4335-4a1c-a5e9-df2444a56e1d-config\") pod \"service-ca-operator-57b5b68b8-smrgr\" (UID: \"afbc552d-4335-4a1c-a5e9-df2444a56e1d\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-smrgr"
May 20 15:34:07.732857 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.732835    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/afbc552d-4335-4a1c-a5e9-df2444a56e1d-serving-cert\") pod \"service-ca-operator-57b5b68b8-smrgr\" (UID: \"afbc552d-4335-4a1c-a5e9-df2444a56e1d\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-smrgr"
May 20 15:34:07.740818 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.740791    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq8fs\" (UniqueName: \"kubernetes.io/projected/afbc552d-4335-4a1c-a5e9-df2444a56e1d-kube-api-access-gq8fs\") pod \"service-ca-operator-57b5b68b8-smrgr\" (UID: \"afbc552d-4335-4a1c-a5e9-df2444a56e1d\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-smrgr"
May 20 15:34:07.778314 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.777265    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-7crcg"]
May 20 15:34:07.781665 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:34:07.781635    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f685e9_2b4e_47a2_b2e0_dd6158968cb3.slice/crio-6ae90d31ad669e7a69d6671e8999c3f54ae88f73463a16b317a4bf619b0ed864 WatchSource:0}: Error finding container 6ae90d31ad669e7a69d6671e8999c3f54ae88f73463a16b317a4bf619b0ed864: Status 404 returned error can't find the container with id 6ae90d31ad669e7a69d6671e8999c3f54ae88f73463a16b317a4bf619b0ed864
May 20 15:34:07.835299 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.835270    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-smrgr"
May 20 15:34:07.977815 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:07.977784    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-smrgr"]
May 20 15:34:07.981045 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:34:07.980986    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafbc552d_4335_4a1c_a5e9_df2444a56e1d.slice/crio-d81d1c98b7b8d5e4b853d87770c250f0707132f3554090cab426c598d636ddc3 WatchSource:0}: Error finding container d81d1c98b7b8d5e4b853d87770c250f0707132f3554090cab426c598d636ddc3: Status 404 returned error can't find the container with id d81d1c98b7b8d5e4b853d87770c250f0707132f3554090cab426c598d636ddc3
May 20 15:34:08.125174 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:08.125136    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-7crcg" event={"ID":"74f685e9-2b4e-47a2-b2e0-dd6158968cb3","Type":"ContainerStarted","Data":"91ff35e8ced52e95a339444f84d8afa9098361b73a4c30a0090628ec4467e6ba"}
May 20 15:34:08.125174 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:08.125178    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-7crcg" event={"ID":"74f685e9-2b4e-47a2-b2e0-dd6158968cb3","Type":"ContainerStarted","Data":"6ae90d31ad669e7a69d6671e8999c3f54ae88f73463a16b317a4bf619b0ed864"}
May 20 15:34:08.126157 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:08.126133    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-smrgr" event={"ID":"afbc552d-4335-4a1c-a5e9-df2444a56e1d","Type":"ContainerStarted","Data":"d81d1c98b7b8d5e4b853d87770c250f0707132f3554090cab426c598d636ddc3"}
May 20 15:34:08.151011 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:08.150901    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-7crcg" podStartSLOduration=1.150883135 podStartE2EDuration="1.150883135s" podCreationTimestamp="2026-05-20 15:34:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:34:08.149726699 +0000 UTC m=+157.088568256" watchObservedRunningTime="2026-05-20 15:34:08.150883135 +0000 UTC m=+157.089724693"
May 20 15:34:08.593944 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:08.593905    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-dbg6m"]
May 20 15:34:08.596471 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:08.596450    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-dbg6m"
May 20 15:34:08.600870 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:08.600842    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
May 20 15:34:08.600992 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:08.600845    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jf7gg\""
May 20 15:34:08.601467 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:08.601441    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
May 20 15:34:08.606058 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:08.605820    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-dbg6m"]
May 20 15:34:08.638930 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:08.638810    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfjrf\" (UniqueName: \"kubernetes.io/projected/1ba2f334-7a2e-4fff-a6e7-7590c25e1dae-kube-api-access-gfjrf\") pod \"migrator-b985f5d89-dbg6m\" (UID: \"1ba2f334-7a2e-4fff-a6e7-7590c25e1dae\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-dbg6m"
May 20 15:34:08.740013 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:08.739975    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gfjrf\" (UniqueName: \"kubernetes.io/projected/1ba2f334-7a2e-4fff-a6e7-7590c25e1dae-kube-api-access-gfjrf\") pod \"migrator-b985f5d89-dbg6m\" (UID: \"1ba2f334-7a2e-4fff-a6e7-7590c25e1dae\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-dbg6m"
May 20 15:34:08.750370 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:08.750335    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfjrf\" (UniqueName: \"kubernetes.io/projected/1ba2f334-7a2e-4fff-a6e7-7590c25e1dae-kube-api-access-gfjrf\") pod \"migrator-b985f5d89-dbg6m\" (UID: \"1ba2f334-7a2e-4fff-a6e7-7590c25e1dae\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-dbg6m"
May 20 15:34:08.909226 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:08.909141    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-dbg6m"
May 20 15:34:08.916279 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:08.916239    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj" podUID="94373970-3fe5-4bbf-a05a-f4913b27e8d0"
May 20 15:34:08.939598 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:08.939535    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-666497c696-xxwnj" podUID="c010265c-2cd5-4e4f-9fd5-f75755dc1419"
May 20 15:34:08.963103 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:08.962981    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-w8bsv" podUID="9bcdce67-2f01-4c5e-a20f-c976159f8683"
May 20 15:34:08.985728 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:08.985662    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-szz5l" podUID="fbac1f01-44d4-4042-a375-5a6a14f94d47"
May 20 15:34:09.059788 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:09.059719    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-dbg6m"]
May 20 15:34:09.062062 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:34:09.062013    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba2f334_7a2e_4fff_a6e7_7590c25e1dae.slice/crio-23eeea376f56d6d81a75989880d69353e89d326be3ae04d3d69a1116d848d517 WatchSource:0}: Error finding container 23eeea376f56d6d81a75989880d69353e89d326be3ae04d3d69a1116d848d517: Status 404 returned error can't find the container with id 23eeea376f56d6d81a75989880d69353e89d326be3ae04d3d69a1116d848d517
May 20 15:34:09.130613 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:09.130558    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-dbg6m" event={"ID":"1ba2f334-7a2e-4fff-a6e7-7590c25e1dae","Type":"ContainerStarted","Data":"23eeea376f56d6d81a75989880d69353e89d326be3ae04d3d69a1116d848d517"}
May 20 15:34:09.131091 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:09.130618    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj"
May 20 15:34:09.131091 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:09.130668    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:34:09.131091 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:09.130666    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-szz5l"
May 20 15:34:09.131291 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:09.131161    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w8bsv"
May 20 15:34:09.577500 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:09.577474    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vkh2n_abd4b634-19df-4a65-b43b-f3cd86482afe/dns-node-resolver/0.log"
May 20 15:34:10.577110 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:10.577071    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l24nh_2fe4adfc-e1dd-4a70-a161-f0d525f3ca3b/node-ca/0.log"
May 20 15:34:10.662821 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:10.662772    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-skk8q" podUID="3022a27a-6112-46ee-a54b-c5af5bee2a1f"
May 20 15:34:11.137115 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:11.137067    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-dbg6m" event={"ID":"1ba2f334-7a2e-4fff-a6e7-7590c25e1dae","Type":"ContainerStarted","Data":"b756b50cd37a9786abed7eab2d949b8604ccc274bea7a4d224a3859e8af95167"}
May 20 15:34:11.137115 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:11.137119    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-dbg6m" event={"ID":"1ba2f334-7a2e-4fff-a6e7-7590c25e1dae","Type":"ContainerStarted","Data":"ce00357917588b73a5f82df4d86ac8126f27d7a0e6e3098d1ce09470214d225b"}
May 20 15:34:11.138360 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:11.138334    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-smrgr" event={"ID":"afbc552d-4335-4a1c-a5e9-df2444a56e1d","Type":"ContainerStarted","Data":"889d3e4b979520d1895b44b75f0eab0b4e785ef33f1a9a7b273ac1fb882b0174"}
May 20 15:34:11.155741 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:11.155679    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-dbg6m" podStartSLOduration=2.053506235 podStartE2EDuration="3.155660797s" podCreationTimestamp="2026-05-20 15:34:08 +0000 UTC" firstStartedPulling="2026-05-20 15:34:09.063955895 +0000 UTC m=+158.002797430" lastFinishedPulling="2026-05-20 15:34:10.166110456 +0000 UTC m=+159.104951992" observedRunningTime="2026-05-20 15:34:11.15451383 +0000 UTC m=+160.093355392" watchObservedRunningTime="2026-05-20 15:34:11.155660797 +0000 UTC m=+160.094502354"
May 20 15:34:11.159811 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:11.159784    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e984d532-c6fd-4d79-8b16-07fc216f278d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-c85d6\" (UID: \"e984d532-c6fd-4d79-8b16-07fc216f278d\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6"
May 20 15:34:11.159952 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:11.159920    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 20 15:34:11.160003 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:11.159980    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e984d532-c6fd-4d79-8b16-07fc216f278d-cluster-monitoring-operator-tls podName:e984d532-c6fd-4d79-8b16-07fc216f278d nodeName:}" failed. No retries permitted until 2026-05-20 15:34:19.159963444 +0000 UTC m=+168.098804982 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e984d532-c6fd-4d79-8b16-07fc216f278d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-c85d6" (UID: "e984d532-c6fd-4d79-8b16-07fc216f278d") : secret "cluster-monitoring-operator-tls" not found
May 20 15:34:11.174513 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:11.174459    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-smrgr" podStartSLOduration=1.990190763 podStartE2EDuration="4.174443389s" podCreationTimestamp="2026-05-20 15:34:07 +0000 UTC" firstStartedPulling="2026-05-20 15:34:07.983172827 +0000 UTC m=+156.922014378" lastFinishedPulling="2026-05-20 15:34:10.167425346 +0000 UTC m=+159.106267004" observedRunningTime="2026-05-20 15:34:11.173059202 +0000 UTC m=+160.111900754" watchObservedRunningTime="2026-05-20 15:34:11.174443389 +0000 UTC m=+160.113284945"
May 20 15:34:11.362579 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:11.362541    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dba48cd-33a2-4fde-a534-38579b96efe0-samples-operator-tls\") pod \"cluster-samples-operator-697587599-x844v\" (UID: \"0dba48cd-33a2-4fde-a534-38579b96efe0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x844v"
May 20 15:34:11.362729 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:11.362662    2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 20 15:34:11.362729 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:11.362726    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dba48cd-33a2-4fde-a534-38579b96efe0-samples-operator-tls podName:0dba48cd-33a2-4fde-a534-38579b96efe0 nodeName:}" failed. No retries permitted until 2026-05-20 15:34:19.362706689 +0000 UTC m=+168.301548231 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0dba48cd-33a2-4fde-a534-38579b96efe0-samples-operator-tls") pod "cluster-samples-operator-697587599-x844v" (UID: "0dba48cd-33a2-4fde-a534-38579b96efe0") : secret "samples-operator-tls" not found
May 20 15:34:13.786624 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:13.786594    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-cdtrw"]
May 20 15:34:13.788606 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:13.788590    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-cdtrw"
May 20 15:34:13.791084 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:13.791055    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-4nrfp\""
May 20 15:34:13.791212 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:13.791097    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
May 20 15:34:13.791750 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:13.791735    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
May 20 15:34:13.791824 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:13.791816    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
May 20 15:34:13.791866 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:13.791841    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
May 20 15:34:13.801949 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:13.801923    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-cdtrw"]
May 20 15:34:13.885822 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:13.885782    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls\") pod \"image-registry-666497c696-xxwnj\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") " pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:34:13.885994 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:13.885831    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkqms\" (UniqueName: \"kubernetes.io/projected/b95cbbb6-fdd4-4783-b884-a4fe28081923-kube-api-access-pkqms\") pod \"service-ca-67d667c7d-cdtrw\" (UID: \"b95cbbb6-fdd4-4783-b884-a4fe28081923\") " pod="openshift-service-ca/service-ca-67d667c7d-cdtrw"
May 20 15:34:13.885994 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:13.885933    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b95cbbb6-fdd4-4783-b884-a4fe28081923-signing-key\") pod \"service-ca-67d667c7d-cdtrw\" (UID: \"b95cbbb6-fdd4-4783-b884-a4fe28081923\") " pod="openshift-service-ca/service-ca-67d667c7d-cdtrw"
May 20 15:34:13.885994 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:13.885941    2568 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 15:34:13.885994 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:13.885961    2568 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666497c696-xxwnj: secret "image-registry-tls" not found
May 20 15:34:13.886159 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:13.886008    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls podName:c010265c-2cd5-4e4f-9fd5-f75755dc1419 nodeName:}" failed. No retries permitted until 2026-05-20 15:36:15.885993599 +0000 UTC m=+284.824835135 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls") pod "image-registry-666497c696-xxwnj" (UID: "c010265c-2cd5-4e4f-9fd5-f75755dc1419") : secret "image-registry-tls" not found
May 20 15:34:13.886159 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:13.886042    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b95cbbb6-fdd4-4783-b884-a4fe28081923-signing-cabundle\") pod \"service-ca-67d667c7d-cdtrw\" (UID: \"b95cbbb6-fdd4-4783-b884-a4fe28081923\") " pod="openshift-service-ca/service-ca-67d667c7d-cdtrw"
May 20 15:34:13.886159 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:13.886093    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-2fdvj\" (UID: \"94373970-3fe5-4bbf-a05a-f4913b27e8d0\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj"
May 20 15:34:13.886257 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:13.886174    2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 20 15:34:13.886257 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:13.886204    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert podName:94373970-3fe5-4bbf-a05a-f4913b27e8d0 nodeName:}" failed. No retries permitted until 2026-05-20 15:36:15.886196401 +0000 UTC m=+284.825037937 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-2fdvj" (UID: "94373970-3fe5-4bbf-a05a-f4913b27e8d0") : secret "networking-console-plugin-cert" not found
May 20 15:34:13.987329 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:13.987292    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pkqms\" (UniqueName: \"kubernetes.io/projected/b95cbbb6-fdd4-4783-b884-a4fe28081923-kube-api-access-pkqms\") pod \"service-ca-67d667c7d-cdtrw\" (UID: \"b95cbbb6-fdd4-4783-b884-a4fe28081923\") " pod="openshift-service-ca/service-ca-67d667c7d-cdtrw"
May 20 15:34:13.987329 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:13.987329    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b95cbbb6-fdd4-4783-b884-a4fe28081923-signing-key\") pod \"service-ca-67d667c7d-cdtrw\" (UID: \"b95cbbb6-fdd4-4783-b884-a4fe28081923\") " pod="openshift-service-ca/service-ca-67d667c7d-cdtrw"
May 20 15:34:13.987540 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:13.987355    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert\") pod \"ingress-canary-szz5l\" (UID: \"fbac1f01-44d4-4042-a375-5a6a14f94d47\") " pod="openshift-ingress-canary/ingress-canary-szz5l"
May 20 15:34:13.987540 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:13.987409    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls\") pod \"dns-default-w8bsv\" (UID: \"9bcdce67-2f01-4c5e-a20f-c976159f8683\") " pod="openshift-dns/dns-default-w8bsv"
May 20 15:34:13.987540 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:13.987427    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b95cbbb6-fdd4-4783-b884-a4fe28081923-signing-cabundle\") pod \"service-ca-67d667c7d-cdtrw\" (UID: \"b95cbbb6-fdd4-4783-b884-a4fe28081923\") " pod="openshift-service-ca/service-ca-67d667c7d-cdtrw"
May 20 15:34:13.987638 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:13.987538    2568 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 15:34:13.987638 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:13.987546    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 15:34:13.987638 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:13.987606    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert podName:fbac1f01-44d4-4042-a375-5a6a14f94d47 nodeName:}" failed. No retries permitted until 2026-05-20 15:36:15.987589859 +0000 UTC m=+284.926431395 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert") pod "ingress-canary-szz5l" (UID: "fbac1f01-44d4-4042-a375-5a6a14f94d47") : secret "canary-serving-cert" not found
May 20 15:34:13.987638 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:13.987622    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls podName:9bcdce67-2f01-4c5e-a20f-c976159f8683 nodeName:}" failed. No retries permitted until 2026-05-20 15:36:15.987616278 +0000 UTC m=+284.926457812 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls") pod "dns-default-w8bsv" (UID: "9bcdce67-2f01-4c5e-a20f-c976159f8683") : secret "dns-default-metrics-tls" not found
May 20 15:34:13.988142 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:13.988125    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b95cbbb6-fdd4-4783-b884-a4fe28081923-signing-cabundle\") pod \"service-ca-67d667c7d-cdtrw\" (UID: \"b95cbbb6-fdd4-4783-b884-a4fe28081923\") " pod="openshift-service-ca/service-ca-67d667c7d-cdtrw"
May 20 15:34:13.989940 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:13.989925    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b95cbbb6-fdd4-4783-b884-a4fe28081923-signing-key\") pod \"service-ca-67d667c7d-cdtrw\" (UID: \"b95cbbb6-fdd4-4783-b884-a4fe28081923\") " pod="openshift-service-ca/service-ca-67d667c7d-cdtrw"
May 20 15:34:13.996268 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:13.996247    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkqms\" (UniqueName: \"kubernetes.io/projected/b95cbbb6-fdd4-4783-b884-a4fe28081923-kube-api-access-pkqms\") pod \"service-ca-67d667c7d-cdtrw\" (UID: \"b95cbbb6-fdd4-4783-b884-a4fe28081923\") " pod="openshift-service-ca/service-ca-67d667c7d-cdtrw"
May 20 15:34:14.097769 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:14.097736    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-cdtrw"
May 20 15:34:14.242406 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:14.242379    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-cdtrw"]
May 20 15:34:14.244187 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:34:14.244157    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95cbbb6_fdd4_4783_b884_a4fe28081923.slice/crio-950c3dc774a0a9c0c2225f158ae6ccea4e4c69643ee6ab12cdfa0e756d085fd8 WatchSource:0}: Error finding container 950c3dc774a0a9c0c2225f158ae6ccea4e4c69643ee6ab12cdfa0e756d085fd8: Status 404 returned error can't find the container with id 950c3dc774a0a9c0c2225f158ae6ccea4e4c69643ee6ab12cdfa0e756d085fd8
May 20 15:34:15.150623 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:15.150584    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-cdtrw" event={"ID":"b95cbbb6-fdd4-4783-b884-a4fe28081923","Type":"ContainerStarted","Data":"578a7e8dd72176c34a26ca9840ee07e816291a1c136f568d83dac1a1f5ca8764"}
May 20 15:34:15.151010 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:15.150631    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-cdtrw" event={"ID":"b95cbbb6-fdd4-4783-b884-a4fe28081923","Type":"ContainerStarted","Data":"950c3dc774a0a9c0c2225f158ae6ccea4e4c69643ee6ab12cdfa0e756d085fd8"}
May 20 15:34:15.168901 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:15.168840    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-67d667c7d-cdtrw" podStartSLOduration=2.168820187 podStartE2EDuration="2.168820187s" podCreationTimestamp="2026-05-20 15:34:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:34:15.168242941 +0000 UTC m=+164.107084497" watchObservedRunningTime="2026-05-20 15:34:15.168820187 +0000 UTC m=+164.107661745"
May 20 15:34:19.236859 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:19.236770    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e984d532-c6fd-4d79-8b16-07fc216f278d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-c85d6\" (UID: \"e984d532-c6fd-4d79-8b16-07fc216f278d\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6"
May 20 15:34:19.237242 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:19.236902    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 20 15:34:19.237242 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:19.236999    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e984d532-c6fd-4d79-8b16-07fc216f278d-cluster-monitoring-operator-tls podName:e984d532-c6fd-4d79-8b16-07fc216f278d nodeName:}" failed. No retries permitted until 2026-05-20 15:34:35.236982526 +0000 UTC m=+184.175824066 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e984d532-c6fd-4d79-8b16-07fc216f278d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-c85d6" (UID: "e984d532-c6fd-4d79-8b16-07fc216f278d") : secret "cluster-monitoring-operator-tls" not found
May 20 15:34:19.439311 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:19.439267    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dba48cd-33a2-4fde-a534-38579b96efe0-samples-operator-tls\") pod \"cluster-samples-operator-697587599-x844v\" (UID: \"0dba48cd-33a2-4fde-a534-38579b96efe0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x844v"
May 20 15:34:19.441754 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:19.441731    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dba48cd-33a2-4fde-a534-38579b96efe0-samples-operator-tls\") pod \"cluster-samples-operator-697587599-x844v\" (UID: \"0dba48cd-33a2-4fde-a534-38579b96efe0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x844v"
May 20 15:34:19.550069 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:19.549969    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x844v"
May 20 15:34:19.703990 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:19.703955    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-x844v"]
May 20 15:34:20.165198 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:20.165162    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x844v" event={"ID":"0dba48cd-33a2-4fde-a534-38579b96efe0","Type":"ContainerStarted","Data":"88cfb3d0fc33a72c056f76cdc075b612a3b44cdac467884f2f846133f5192482"}
May 20 15:34:22.172413 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:22.172379    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x844v" event={"ID":"0dba48cd-33a2-4fde-a534-38579b96efe0","Type":"ContainerStarted","Data":"4d3619517d110226dec3ddbb22a741083a08b0d0880e130052169c99b64a4ed6"}
May 20 15:34:22.172413 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:22.172413    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x844v" event={"ID":"0dba48cd-33a2-4fde-a534-38579b96efe0","Type":"ContainerStarted","Data":"e5bd416e4a9fada522636bc7b7ac442fed40e66b6d84ab1b021cd4bedaba4093"}
May 20 15:34:22.639531 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:22.639491    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:34:35.284235 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.284190    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e984d532-c6fd-4d79-8b16-07fc216f278d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-c85d6\" (UID: \"e984d532-c6fd-4d79-8b16-07fc216f278d\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6"
May 20 15:34:35.286785 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.286764    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e984d532-c6fd-4d79-8b16-07fc216f278d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-c85d6\" (UID: \"e984d532-c6fd-4d79-8b16-07fc216f278d\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6"
May 20 15:34:35.545583 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.545502    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ctbkt\""
May 20 15:34:35.554239 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.554218    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6"
May 20 15:34:35.684436 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.684383    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x844v" podStartSLOduration=31.140379959 podStartE2EDuration="32.684366919s" podCreationTimestamp="2026-05-20 15:34:03 +0000 UTC" firstStartedPulling="2026-05-20 15:34:19.747609521 +0000 UTC m=+168.686451055" lastFinishedPulling="2026-05-20 15:34:21.29159648 +0000 UTC m=+170.230438015" observedRunningTime="2026-05-20 15:34:22.196125939 +0000 UTC m=+171.134967496" watchObservedRunningTime="2026-05-20 15:34:35.684366919 +0000 UTC m=+184.623208526"
May 20 15:34:35.685056 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.685038    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6"]
May 20 15:34:35.687386 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:34:35.687363    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode984d532_c6fd_4d79_8b16_07fc216f278d.slice/crio-6db43cb6e2da1b260ffb9274c7ea4a12d9fde3923932ca5006a7d120e291327f WatchSource:0}: Error finding container 6db43cb6e2da1b260ffb9274c7ea4a12d9fde3923932ca5006a7d120e291327f: Status 404 returned error can't find the container with id 6db43cb6e2da1b260ffb9274c7ea4a12d9fde3923932ca5006a7d120e291327f
May 20 15:34:35.785362 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.785328    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w8wdt"]
May 20 15:34:35.811777 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.811737    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w8wdt"]
May 20 15:34:35.811944 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.811916    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w8wdt"
May 20 15:34:35.814504 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.814485    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s94kz\""
May 20 15:34:35.814690 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.814486    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 20 15:34:35.814896 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.814880    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 20 15:34:35.889530 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.889496    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f44dd61a-10fc-4e25-ba59-ae239dd254e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w8wdt\" (UID: \"f44dd61a-10fc-4e25-ba59-ae239dd254e5\") " pod="openshift-insights/insights-runtime-extractor-w8wdt"
May 20 15:34:35.889530 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.889530    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f44dd61a-10fc-4e25-ba59-ae239dd254e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w8wdt\" (UID: \"f44dd61a-10fc-4e25-ba59-ae239dd254e5\") " pod="openshift-insights/insights-runtime-extractor-w8wdt"
May 20 15:34:35.889725 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.889555    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f44dd61a-10fc-4e25-ba59-ae239dd254e5-crio-socket\") pod \"insights-runtime-extractor-w8wdt\" (UID: \"f44dd61a-10fc-4e25-ba59-ae239dd254e5\") " pod="openshift-insights/insights-runtime-extractor-w8wdt"
May 20 15:34:35.889725 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.889575    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6bfq\" (UniqueName: \"kubernetes.io/projected/f44dd61a-10fc-4e25-ba59-ae239dd254e5-kube-api-access-v6bfq\") pod \"insights-runtime-extractor-w8wdt\" (UID: \"f44dd61a-10fc-4e25-ba59-ae239dd254e5\") " pod="openshift-insights/insights-runtime-extractor-w8wdt"
May 20 15:34:35.889725 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.889702    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f44dd61a-10fc-4e25-ba59-ae239dd254e5-data-volume\") pod \"insights-runtime-extractor-w8wdt\" (UID: \"f44dd61a-10fc-4e25-ba59-ae239dd254e5\") " pod="openshift-insights/insights-runtime-extractor-w8wdt"
May 20 15:34:35.990916 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.990876    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f44dd61a-10fc-4e25-ba59-ae239dd254e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w8wdt\" (UID: \"f44dd61a-10fc-4e25-ba59-ae239dd254e5\") " pod="openshift-insights/insights-runtime-extractor-w8wdt"
May 20 15:34:35.991127 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.990917    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f44dd61a-10fc-4e25-ba59-ae239dd254e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w8wdt\" (UID: \"f44dd61a-10fc-4e25-ba59-ae239dd254e5\") " pod="openshift-insights/insights-runtime-extractor-w8wdt"
May 20 15:34:35.991127 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.990977    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f44dd61a-10fc-4e25-ba59-ae239dd254e5-crio-socket\") pod \"insights-runtime-extractor-w8wdt\" (UID: \"f44dd61a-10fc-4e25-ba59-ae239dd254e5\") " pod="openshift-insights/insights-runtime-extractor-w8wdt"
May 20 15:34:35.991127 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.990999    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v6bfq\" (UniqueName: \"kubernetes.io/projected/f44dd61a-10fc-4e25-ba59-ae239dd254e5-kube-api-access-v6bfq\") pod \"insights-runtime-extractor-w8wdt\" (UID: \"f44dd61a-10fc-4e25-ba59-ae239dd254e5\") " pod="openshift-insights/insights-runtime-extractor-w8wdt"
May 20 15:34:35.991127 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.991106    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f44dd61a-10fc-4e25-ba59-ae239dd254e5-crio-socket\") pod \"insights-runtime-extractor-w8wdt\" (UID: \"f44dd61a-10fc-4e25-ba59-ae239dd254e5\") " pod="openshift-insights/insights-runtime-extractor-w8wdt"
May 20 15:34:35.991332 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.991212    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f44dd61a-10fc-4e25-ba59-ae239dd254e5-data-volume\") pod \"insights-runtime-extractor-w8wdt\" (UID: \"f44dd61a-10fc-4e25-ba59-ae239dd254e5\") " pod="openshift-insights/insights-runtime-extractor-w8wdt"
May 20 15:34:35.991481 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.991464    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f44dd61a-10fc-4e25-ba59-ae239dd254e5-data-volume\") pod \"insights-runtime-extractor-w8wdt\" (UID: \"f44dd61a-10fc-4e25-ba59-ae239dd254e5\") " pod="openshift-insights/insights-runtime-extractor-w8wdt"
May 20 15:34:35.991555 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.991534    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f44dd61a-10fc-4e25-ba59-ae239dd254e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w8wdt\" (UID: \"f44dd61a-10fc-4e25-ba59-ae239dd254e5\") " pod="openshift-insights/insights-runtime-extractor-w8wdt"
May 20 15:34:35.993572 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:35.993550    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f44dd61a-10fc-4e25-ba59-ae239dd254e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w8wdt\" (UID: \"f44dd61a-10fc-4e25-ba59-ae239dd254e5\") " pod="openshift-insights/insights-runtime-extractor-w8wdt"
May 20 15:34:36.001687 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:36.001662    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6bfq\" (UniqueName: \"kubernetes.io/projected/f44dd61a-10fc-4e25-ba59-ae239dd254e5-kube-api-access-v6bfq\") pod \"insights-runtime-extractor-w8wdt\" (UID: \"f44dd61a-10fc-4e25-ba59-ae239dd254e5\") " pod="openshift-insights/insights-runtime-extractor-w8wdt"
May 20 15:34:36.121325 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:36.121231    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w8wdt"
May 20 15:34:36.209822 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:36.209772    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6" event={"ID":"e984d532-c6fd-4d79-8b16-07fc216f278d","Type":"ContainerStarted","Data":"6db43cb6e2da1b260ffb9274c7ea4a12d9fde3923932ca5006a7d120e291327f"}
May 20 15:34:36.258517 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:36.258478    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w8wdt"]
May 20 15:34:36.261421 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:34:36.261390    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44dd61a_10fc_4e25_ba59_ae239dd254e5.slice/crio-eea8757a204befb7aa1ae3ffd6422d70547c06dee064e40269547f3097d4d9bc WatchSource:0}: Error finding container eea8757a204befb7aa1ae3ffd6422d70547c06dee064e40269547f3097d4d9bc: Status 404 returned error can't find the container with id eea8757a204befb7aa1ae3ffd6422d70547c06dee064e40269547f3097d4d9bc
May 20 15:34:37.217852 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:37.217811    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w8wdt" event={"ID":"f44dd61a-10fc-4e25-ba59-ae239dd254e5","Type":"ContainerStarted","Data":"70d07c27394ffe838fcef50d5b85ec1fa866b244f357b00c6321da73ad4b71e1"}
May 20 15:34:37.218277 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:37.217860    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w8wdt" event={"ID":"f44dd61a-10fc-4e25-ba59-ae239dd254e5","Type":"ContainerStarted","Data":"eea8757a204befb7aa1ae3ffd6422d70547c06dee064e40269547f3097d4d9bc"}
May 20 15:34:38.225133 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:38.225013    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w8wdt" event={"ID":"f44dd61a-10fc-4e25-ba59-ae239dd254e5","Type":"ContainerStarted","Data":"ad681b8996df04e032c560d60450a831f18f5a535d8f1af31ba768d1b2031b20"}
May 20 15:34:38.226230 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:38.226195    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6" event={"ID":"e984d532-c6fd-4d79-8b16-07fc216f278d","Type":"ContainerStarted","Data":"42dc87f5d40de39daf2a7157d37d659aa49c590f06605f67cf1ce4401cffb660"}
May 20 15:34:38.248666 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:38.248614    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-c85d6" podStartSLOduration=33.078665416 podStartE2EDuration="35.248600638s" podCreationTimestamp="2026-05-20 15:34:03 +0000 UTC" firstStartedPulling="2026-05-20 15:34:35.689353338 +0000 UTC m=+184.628194888" lastFinishedPulling="2026-05-20 15:34:37.859288575 +0000 UTC m=+186.798130110" observedRunningTime="2026-05-20 15:34:38.247227888 +0000 UTC m=+187.186069444" watchObservedRunningTime="2026-05-20 15:34:38.248600638 +0000 UTC m=+187.187442193"
May 20 15:34:40.232655 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.232620    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w8wdt" event={"ID":"f44dd61a-10fc-4e25-ba59-ae239dd254e5","Type":"ContainerStarted","Data":"5b34b123f962319dd96e1ed49fc0f56b33da606cd06497cbc3fb49502b67d376"}
May 20 15:34:40.251665 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.251609    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w8wdt" podStartSLOduration=2.4397470390000002 podStartE2EDuration="5.251594923s" podCreationTimestamp="2026-05-20 15:34:35 +0000 UTC" firstStartedPulling="2026-05-20 15:34:36.394667779 +0000 UTC m=+185.333509314" lastFinishedPulling="2026-05-20 15:34:39.206515645 +0000 UTC m=+188.145357198" observedRunningTime="2026-05-20 15:34:40.250995049 +0000 UTC m=+189.189836605" watchObservedRunningTime="2026-05-20 15:34:40.251594923 +0000 UTC m=+189.190436480"
May 20 15:34:40.511457 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.511372    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-4g7z6"]
May 20 15:34:40.513662 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.513642    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-4g7z6"
May 20 15:34:40.516861 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.516827    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 20 15:34:40.516861 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.516850    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vfkm7\""
May 20 15:34:40.517067 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.516863    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 20 15:34:40.517067 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.516850    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 20 15:34:40.523248 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.523225    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-4g7z6"]
May 20 15:34:40.531823 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.531796    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef8af763-035c-4a9a-9f38-91d2dcc95823-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-4g7z6\" (UID: \"ef8af763-035c-4a9a-9f38-91d2dcc95823\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4g7z6"
May 20 15:34:40.531966 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.531836    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef8af763-035c-4a9a-9f38-91d2dcc95823-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-4g7z6\" (UID: \"ef8af763-035c-4a9a-9f38-91d2dcc95823\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4g7z6"
May 20 15:34:40.531966 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.531881    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxxwf\" (UniqueName: \"kubernetes.io/projected/ef8af763-035c-4a9a-9f38-91d2dcc95823-kube-api-access-fxxwf\") pod \"prometheus-operator-7779bbdd78-4g7z6\" (UID: \"ef8af763-035c-4a9a-9f38-91d2dcc95823\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4g7z6"
May 20 15:34:40.532061 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.531968    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef8af763-035c-4a9a-9f38-91d2dcc95823-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-4g7z6\" (UID: \"ef8af763-035c-4a9a-9f38-91d2dcc95823\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4g7z6"
May 20 15:34:40.632330 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.632286    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef8af763-035c-4a9a-9f38-91d2dcc95823-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-4g7z6\" (UID: \"ef8af763-035c-4a9a-9f38-91d2dcc95823\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4g7z6"
May 20 15:34:40.632533 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.632375    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef8af763-035c-4a9a-9f38-91d2dcc95823-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-4g7z6\" (UID: \"ef8af763-035c-4a9a-9f38-91d2dcc95823\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4g7z6"
May 20 15:34:40.632533 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.632411    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef8af763-035c-4a9a-9f38-91d2dcc95823-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-4g7z6\" (UID: \"ef8af763-035c-4a9a-9f38-91d2dcc95823\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4g7z6"
May 20 15:34:40.632533 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.632480    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fxxwf\" (UniqueName: \"kubernetes.io/projected/ef8af763-035c-4a9a-9f38-91d2dcc95823-kube-api-access-fxxwf\") pod \"prometheus-operator-7779bbdd78-4g7z6\" (UID: \"ef8af763-035c-4a9a-9f38-91d2dcc95823\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4g7z6"
May 20 15:34:40.633252 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.633231    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef8af763-035c-4a9a-9f38-91d2dcc95823-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-4g7z6\" (UID: \"ef8af763-035c-4a9a-9f38-91d2dcc95823\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4g7z6"
May 20 15:34:40.635008 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.634978    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef8af763-035c-4a9a-9f38-91d2dcc95823-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-4g7z6\" (UID: \"ef8af763-035c-4a9a-9f38-91d2dcc95823\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4g7z6"
May 20 15:34:40.635008 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.634997    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef8af763-035c-4a9a-9f38-91d2dcc95823-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-4g7z6\" (UID: \"ef8af763-035c-4a9a-9f38-91d2dcc95823\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4g7z6"
May 20 15:34:40.641089 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.641066    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxxwf\" (UniqueName: \"kubernetes.io/projected/ef8af763-035c-4a9a-9f38-91d2dcc95823-kube-api-access-fxxwf\") pod \"prometheus-operator-7779bbdd78-4g7z6\" (UID: \"ef8af763-035c-4a9a-9f38-91d2dcc95823\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-4g7z6"
May 20 15:34:40.823722 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.823686    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-4g7z6"
May 20 15:34:40.954606 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:40.954568    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-4g7z6"]
May 20 15:34:40.957417 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:34:40.957382    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8af763_035c_4a9a_9f38_91d2dcc95823.slice/crio-cf6bd31c55b0aad39269f8117fb0685042715ef086a018d429af86f7716da7c5 WatchSource:0}: Error finding container cf6bd31c55b0aad39269f8117fb0685042715ef086a018d429af86f7716da7c5: Status 404 returned error can't find the container with id cf6bd31c55b0aad39269f8117fb0685042715ef086a018d429af86f7716da7c5
May 20 15:34:41.240408 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:41.240312    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-4g7z6" event={"ID":"ef8af763-035c-4a9a-9f38-91d2dcc95823","Type":"ContainerStarted","Data":"cf6bd31c55b0aad39269f8117fb0685042715ef086a018d429af86f7716da7c5"}
May 20 15:34:42.244621 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:42.244588    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-4g7z6" event={"ID":"ef8af763-035c-4a9a-9f38-91d2dcc95823","Type":"ContainerStarted","Data":"3d37bf385f4592f58c4be83208c0908dbe488e5baa1f0e28503d2541c32f8284"}
May 20 15:34:43.250671 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:43.250634    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-4g7z6" event={"ID":"ef8af763-035c-4a9a-9f38-91d2dcc95823","Type":"ContainerStarted","Data":"ba3bc8d16c7ab1c4dce5236cdee9454cea01545998e7bdd88a6725998b9fbce1"}
May 20 15:34:43.269479 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:43.269408    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-4g7z6" podStartSLOduration=2.070829894 podStartE2EDuration="3.269387644s" podCreationTimestamp="2026-05-20 15:34:40 +0000 UTC" firstStartedPulling="2026-05-20 15:34:40.959312093 +0000 UTC m=+189.898153628" lastFinishedPulling="2026-05-20 15:34:42.157869839 +0000 UTC m=+191.096711378" observedRunningTime="2026-05-20 15:34:43.269086748 +0000 UTC m=+192.207928308" watchObservedRunningTime="2026-05-20 15:34:43.269387644 +0000 UTC m=+192.208229202"
May 20 15:34:44.865584 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.865543    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn"]
May 20 15:34:44.868197 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.868177    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn"
May 20 15:34:44.870430 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.870408    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 20 15:34:44.870520 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.870413    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 20 15:34:44.870686 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.870669    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hxmf7\""
May 20 15:34:44.878667 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.878641    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn"]
May 20 15:34:44.891001 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.890958    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"]
May 20 15:34:44.899762 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.899734    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gd2xt"]
May 20 15:34:44.899971 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.899952    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"
May 20 15:34:44.902203 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.902172    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:44.902562 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.902328    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-vcznn\""
May 20 15:34:44.902562 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.902349    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
May 20 15:34:44.902562 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.902355    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
May 20 15:34:44.902754 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.902592    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
May 20 15:34:44.905657 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.905599    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 20 15:34:44.905858 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.905841    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 20 15:34:44.906099 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.906084    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fklp5\""
May 20 15:34:44.906181 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.906118    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 20 15:34:44.906898 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.906877    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"]
May 20 15:34:44.973295 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.973252    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ddf79fd-adcf-46d4-b6cf-27f541a98e08-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-nfcxl\" (UID: \"9ddf79fd-adcf-46d4-b6cf-27f541a98e08\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"
May 20 15:34:44.973484 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.973299    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ddf79fd-adcf-46d4-b6cf-27f541a98e08-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-nfcxl\" (UID: \"9ddf79fd-adcf-46d4-b6cf-27f541a98e08\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"
May 20 15:34:44.973484 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.973388    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9ddf79fd-adcf-46d4-b6cf-27f541a98e08-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-nfcxl\" (UID: \"9ddf79fd-adcf-46d4-b6cf-27f541a98e08\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"
May 20 15:34:44.973484 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.973445    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6be29376-46fb-484c-87de-2a74cb66ce76-root\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:44.973613 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.973487    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/52f2ef1f-4690-4c87-8acd-ded5750e21e4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-8k4jn\" (UID: \"52f2ef1f-4690-4c87-8acd-ded5750e21e4\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn"
May 20 15:34:44.973613 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.973518    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6be29376-46fb-484c-87de-2a74cb66ce76-node-exporter-accelerators-collector-config\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:44.973613 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.973545    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/52f2ef1f-4690-4c87-8acd-ded5750e21e4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-8k4jn\" (UID: \"52f2ef1f-4690-4c87-8acd-ded5750e21e4\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn"
May 20 15:34:44.973613 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.973575    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6be29376-46fb-484c-87de-2a74cb66ce76-node-exporter-tls\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:44.973815 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.973624    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ddf79fd-adcf-46d4-b6cf-27f541a98e08-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-nfcxl\" (UID: \"9ddf79fd-adcf-46d4-b6cf-27f541a98e08\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"
May 20 15:34:44.973815 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.973673    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxqj2\" (UniqueName: \"kubernetes.io/projected/52f2ef1f-4690-4c87-8acd-ded5750e21e4-kube-api-access-xxqj2\") pod \"openshift-state-metrics-9c959559f-8k4jn\" (UID: \"52f2ef1f-4690-4c87-8acd-ded5750e21e4\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn"
May 20 15:34:44.973815 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.973710    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52f2ef1f-4690-4c87-8acd-ded5750e21e4-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-8k4jn\" (UID: \"52f2ef1f-4690-4c87-8acd-ded5750e21e4\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn"
May 20 15:34:44.973815 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.973738    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89wbl\" (UniqueName: \"kubernetes.io/projected/9ddf79fd-adcf-46d4-b6cf-27f541a98e08-kube-api-access-89wbl\") pod \"kube-state-metrics-54fd6bc5c4-nfcxl\" (UID: \"9ddf79fd-adcf-46d4-b6cf-27f541a98e08\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"
May 20 15:34:44.973815 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.973775    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6be29376-46fb-484c-87de-2a74cb66ce76-sys\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:44.973815 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.973810    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6be29376-46fb-484c-87de-2a74cb66ce76-node-exporter-wtmp\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:44.974100 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.973869    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6be29376-46fb-484c-87de-2a74cb66ce76-metrics-client-ca\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:44.974100 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.973899    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6be29376-46fb-484c-87de-2a74cb66ce76-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:44.974100 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.973959    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6be29376-46fb-484c-87de-2a74cb66ce76-node-exporter-textfile\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:44.974100 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.973987    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbqxb\" (UniqueName: \"kubernetes.io/projected/6be29376-46fb-484c-87de-2a74cb66ce76-kube-api-access-hbqxb\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:44.974100 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:44.974061    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9ddf79fd-adcf-46d4-b6cf-27f541a98e08-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-nfcxl\" (UID: \"9ddf79fd-adcf-46d4-b6cf-27f541a98e08\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"
May 20 15:34:45.075351 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.075316    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6be29376-46fb-484c-87de-2a74cb66ce76-node-exporter-wtmp\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:45.075532 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.075370    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6be29376-46fb-484c-87de-2a74cb66ce76-metrics-client-ca\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:45.075532 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.075398    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6be29376-46fb-484c-87de-2a74cb66ce76-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:45.075532 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.075443    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6be29376-46fb-484c-87de-2a74cb66ce76-node-exporter-textfile\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:45.075532 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.075472    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hbqxb\" (UniqueName: \"kubernetes.io/projected/6be29376-46fb-484c-87de-2a74cb66ce76-kube-api-access-hbqxb\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:45.075532 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.075512    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9ddf79fd-adcf-46d4-b6cf-27f541a98e08-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-nfcxl\" (UID: \"9ddf79fd-adcf-46d4-b6cf-27f541a98e08\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"
May 20 15:34:45.075532 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.075523    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6be29376-46fb-484c-87de-2a74cb66ce76-node-exporter-wtmp\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:45.075847 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.075591    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ddf79fd-adcf-46d4-b6cf-27f541a98e08-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-nfcxl\" (UID: \"9ddf79fd-adcf-46d4-b6cf-27f541a98e08\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"
May 20 15:34:45.075847 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.075628    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ddf79fd-adcf-46d4-b6cf-27f541a98e08-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-nfcxl\" (UID: \"9ddf79fd-adcf-46d4-b6cf-27f541a98e08\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"
May 20 15:34:45.075847 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.075679    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9ddf79fd-adcf-46d4-b6cf-27f541a98e08-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-nfcxl\" (UID: \"9ddf79fd-adcf-46d4-b6cf-27f541a98e08\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"
May 20 15:34:45.075847 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.075708    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6be29376-46fb-484c-87de-2a74cb66ce76-root\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:45.075847 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.075743    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/52f2ef1f-4690-4c87-8acd-ded5750e21e4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-8k4jn\" (UID: \"52f2ef1f-4690-4c87-8acd-ded5750e21e4\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn"
May 20 15:34:45.075847 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.075771    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6be29376-46fb-484c-87de-2a74cb66ce76-node-exporter-accelerators-collector-config\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:45.075847 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.075799    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/52f2ef1f-4690-4c87-8acd-ded5750e21e4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-8k4jn\" (UID: \"52f2ef1f-4690-4c87-8acd-ded5750e21e4\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn"
May 20 15:34:45.076199 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.076042    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6be29376-46fb-484c-87de-2a74cb66ce76-metrics-client-ca\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:45.076266 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.076178    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6be29376-46fb-484c-87de-2a74cb66ce76-node-exporter-textfile\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:45.076319 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.076265    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9ddf79fd-adcf-46d4-b6cf-27f541a98e08-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-nfcxl\" (UID: \"9ddf79fd-adcf-46d4-b6cf-27f541a98e08\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"
May 20 15:34:45.076370 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.076319    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9ddf79fd-adcf-46d4-b6cf-27f541a98e08-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-nfcxl\" (UID: \"9ddf79fd-adcf-46d4-b6cf-27f541a98e08\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"
May 20 15:34:45.076424 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:45.076407    2568 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
May 20 15:34:45.076470 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:45.076458    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ddf79fd-adcf-46d4-b6cf-27f541a98e08-kube-state-metrics-tls podName:9ddf79fd-adcf-46d4-b6cf-27f541a98e08 nodeName:}" failed. No retries permitted until 2026-05-20 15:34:45.576441812 +0000 UTC m=+194.515283351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/9ddf79fd-adcf-46d4-b6cf-27f541a98e08-kube-state-metrics-tls") pod "kube-state-metrics-54fd6bc5c4-nfcxl" (UID: "9ddf79fd-adcf-46d4-b6cf-27f541a98e08") : secret "kube-state-metrics-tls" not found
May 20 15:34:45.076703 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:45.076620    2568 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
May 20 15:34:45.076703 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:45.076655    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52f2ef1f-4690-4c87-8acd-ded5750e21e4-openshift-state-metrics-tls podName:52f2ef1f-4690-4c87-8acd-ded5750e21e4 nodeName:}" failed. No retries permitted until 2026-05-20 15:34:45.576643816 +0000 UTC m=+194.515485355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/52f2ef1f-4690-4c87-8acd-ded5750e21e4-openshift-state-metrics-tls") pod "openshift-state-metrics-9c959559f-8k4jn" (UID: "52f2ef1f-4690-4c87-8acd-ded5750e21e4") : secret "openshift-state-metrics-tls" not found
May 20 15:34:45.076821 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.076699    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6be29376-46fb-484c-87de-2a74cb66ce76-root\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:45.076821 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.076788    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6be29376-46fb-484c-87de-2a74cb66ce76-node-exporter-tls\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:45.076934 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.076884    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ddf79fd-adcf-46d4-b6cf-27f541a98e08-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-nfcxl\" (UID: \"9ddf79fd-adcf-46d4-b6cf-27f541a98e08\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"
May 20 15:34:45.076987 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.076933    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xxqj2\" (UniqueName: \"kubernetes.io/projected/52f2ef1f-4690-4c87-8acd-ded5750e21e4-kube-api-access-xxqj2\") pod \"openshift-state-metrics-9c959559f-8k4jn\" (UID: \"52f2ef1f-4690-4c87-8acd-ded5750e21e4\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn"
May 20 15:34:45.076987 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.076967    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52f2ef1f-4690-4c87-8acd-ded5750e21e4-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-8k4jn\" (UID: \"52f2ef1f-4690-4c87-8acd-ded5750e21e4\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn"
May 20 15:34:45.077128 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.077007    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-89wbl\" (UniqueName: \"kubernetes.io/projected/9ddf79fd-adcf-46d4-b6cf-27f541a98e08-kube-api-access-89wbl\") pod \"kube-state-metrics-54fd6bc5c4-nfcxl\" (UID: \"9ddf79fd-adcf-46d4-b6cf-27f541a98e08\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"
May 20 15:34:45.077128 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.077055    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6be29376-46fb-484c-87de-2a74cb66ce76-node-exporter-accelerators-collector-config\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:45.077128 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.077072    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6be29376-46fb-484c-87de-2a74cb66ce76-sys\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:45.077278 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.077218    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6be29376-46fb-484c-87de-2a74cb66ce76-sys\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:45.077854 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.077832    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ddf79fd-adcf-46d4-b6cf-27f541a98e08-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-nfcxl\" (UID: \"9ddf79fd-adcf-46d4-b6cf-27f541a98e08\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"
May 20 15:34:45.078063 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.078043    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52f2ef1f-4690-4c87-8acd-ded5750e21e4-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-8k4jn\" (UID: \"52f2ef1f-4690-4c87-8acd-ded5750e21e4\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn"
May 20 15:34:45.078539 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.078511    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6be29376-46fb-484c-87de-2a74cb66ce76-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:45.078846 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.078823    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/52f2ef1f-4690-4c87-8acd-ded5750e21e4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-8k4jn\" (UID: \"52f2ef1f-4690-4c87-8acd-ded5750e21e4\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn"
May 20 15:34:45.079471 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.079452    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ddf79fd-adcf-46d4-b6cf-27f541a98e08-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-nfcxl\" (UID: \"9ddf79fd-adcf-46d4-b6cf-27f541a98e08\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"
May 20 15:34:45.080177 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.080158    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6be29376-46fb-484c-87de-2a74cb66ce76-node-exporter-tls\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:45.088541 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.088512    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxqj2\" (UniqueName: \"kubernetes.io/projected/52f2ef1f-4690-4c87-8acd-ded5750e21e4-kube-api-access-xxqj2\") pod \"openshift-state-metrics-9c959559f-8k4jn\" (UID: \"52f2ef1f-4690-4c87-8acd-ded5750e21e4\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn"
May 20 15:34:45.089321 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.089298    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbqxb\" (UniqueName: \"kubernetes.io/projected/6be29376-46fb-484c-87de-2a74cb66ce76-kube-api-access-hbqxb\") pod \"node-exporter-gd2xt\" (UID: \"6be29376-46fb-484c-87de-2a74cb66ce76\") " pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:45.091382 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.091359    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89wbl\" (UniqueName: \"kubernetes.io/projected/9ddf79fd-adcf-46d4-b6cf-27f541a98e08-kube-api-access-89wbl\") pod \"kube-state-metrics-54fd6bc5c4-nfcxl\" (UID: \"9ddf79fd-adcf-46d4-b6cf-27f541a98e08\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"
May 20 15:34:45.218486 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.218398    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gd2xt"
May 20 15:34:45.226542 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:34:45.226513    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be29376_46fb_484c_87de_2a74cb66ce76.slice/crio-5d38160666fb044ae9f33ea222178f040b424482669c121aca8c1cf54d78c044 WatchSource:0}: Error finding container 5d38160666fb044ae9f33ea222178f040b424482669c121aca8c1cf54d78c044: Status 404 returned error can't find the container with id 5d38160666fb044ae9f33ea222178f040b424482669c121aca8c1cf54d78c044
May 20 15:34:45.257595 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.257559    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gd2xt" event={"ID":"6be29376-46fb-484c-87de-2a74cb66ce76","Type":"ContainerStarted","Data":"5d38160666fb044ae9f33ea222178f040b424482669c121aca8c1cf54d78c044"}
May 20 15:34:45.580483 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.580451    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/52f2ef1f-4690-4c87-8acd-ded5750e21e4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-8k4jn\" (UID: \"52f2ef1f-4690-4c87-8acd-ded5750e21e4\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn"
May 20 15:34:45.580662 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.580576    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ddf79fd-adcf-46d4-b6cf-27f541a98e08-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-nfcxl\" (UID: \"9ddf79fd-adcf-46d4-b6cf-27f541a98e08\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"
May 20 15:34:45.583286 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.583256    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ddf79fd-adcf-46d4-b6cf-27f541a98e08-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-nfcxl\" (UID: \"9ddf79fd-adcf-46d4-b6cf-27f541a98e08\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"
May 20 15:34:45.583398 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.583256    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/52f2ef1f-4690-4c87-8acd-ded5750e21e4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-8k4jn\" (UID: \"52f2ef1f-4690-4c87-8acd-ded5750e21e4\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn"
May 20 15:34:45.780173 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.780141    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn"
May 20 15:34:45.813148 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.813113    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"
May 20 15:34:45.954664 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:45.954630    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn"]
May 20 15:34:46.001125 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:34:46.001080    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ddf79fd_adcf_46d4_b6cf_27f541a98e08.slice/crio-3e8c5bb1a32cbc5d29bf562062283dc6da4268c47b94a0baa54f4e29252fa078 WatchSource:0}: Error finding container 3e8c5bb1a32cbc5d29bf562062283dc6da4268c47b94a0baa54f4e29252fa078: Status 404 returned error can't find the container with id 3e8c5bb1a32cbc5d29bf562062283dc6da4268c47b94a0baa54f4e29252fa078
May 20 15:34:46.001777 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:46.001751    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl"]
May 20 15:34:46.262174 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:46.262122    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl" event={"ID":"9ddf79fd-adcf-46d4-b6cf-27f541a98e08","Type":"ContainerStarted","Data":"3e8c5bb1a32cbc5d29bf562062283dc6da4268c47b94a0baa54f4e29252fa078"}
May 20 15:34:46.264089 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:46.264060    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn" event={"ID":"52f2ef1f-4690-4c87-8acd-ded5750e21e4","Type":"ContainerStarted","Data":"0ebf88204ea69715cb62636925d31e01674ceda0f39fc7594d711a19961f43e2"}
May 20 15:34:46.264226 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:46.264095    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn" event={"ID":"52f2ef1f-4690-4c87-8acd-ded5750e21e4","Type":"ContainerStarted","Data":"7f7eb294a1c0e70c53180adfdf7de86c810d9818787eb39dc3be4587000bcce9"}
May 20 15:34:46.264226 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:46.264108    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn" event={"ID":"52f2ef1f-4690-4c87-8acd-ded5750e21e4","Type":"ContainerStarted","Data":"da1831f779223271fade8c75e237bc0ee0220d86d786ae6a20f2d43e70ee5254"}
May 20 15:34:47.269133 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:47.269075    2568 generic.go:358] "Generic (PLEG): container finished" podID="6be29376-46fb-484c-87de-2a74cb66ce76" containerID="fbd7c3bf895a5f734016d3b4a97632a84a8c3fbc517682914c8cdfb2f7d4efe9" exitCode=0
May 20 15:34:47.269601 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:47.269164    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gd2xt" event={"ID":"6be29376-46fb-484c-87de-2a74cb66ce76","Type":"ContainerDied","Data":"fbd7c3bf895a5f734016d3b4a97632a84a8c3fbc517682914c8cdfb2f7d4efe9"}
May 20 15:34:48.273866 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:48.273824    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gd2xt" event={"ID":"6be29376-46fb-484c-87de-2a74cb66ce76","Type":"ContainerStarted","Data":"d220dd9e3c15d412911d6855e8853a41b9d7d4a0ee59c8062f3deebe46f2f4da"}
May 20 15:34:48.274343 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:48.273903    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gd2xt" event={"ID":"6be29376-46fb-484c-87de-2a74cb66ce76","Type":"ContainerStarted","Data":"bb85d43a54aedd68b1579068c5f62ca7d907ba71604ac68480cf84838e771ac2"}
May 20 15:34:48.276189 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:48.276160    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn" event={"ID":"52f2ef1f-4690-4c87-8acd-ded5750e21e4","Type":"ContainerStarted","Data":"1ff04915f061b39192f55ad92174b84fff7a459b50db68a0c56e7eeafaea1cf6"}
May 20 15:34:48.278410 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:48.278380    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl" event={"ID":"9ddf79fd-adcf-46d4-b6cf-27f541a98e08","Type":"ContainerStarted","Data":"fcda602e6ed2412383015b1003b9f6460a891676473da1d396405d98d46fdf2e"}
May 20 15:34:48.278520 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:48.278414    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl" event={"ID":"9ddf79fd-adcf-46d4-b6cf-27f541a98e08","Type":"ContainerStarted","Data":"9af80a1420fc41de5331bbb6ccbfecf0d1bca521554b88291f4d97b57b792a00"}
May 20 15:34:48.278520 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:48.278424    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl" event={"ID":"9ddf79fd-adcf-46d4-b6cf-27f541a98e08","Type":"ContainerStarted","Data":"dea9f27d6e9192138f04828887b0876071684fb56088b51a9c27cded35424a95"}
May 20 15:34:48.295139 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:48.295090    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gd2xt" podStartSLOduration=3.101365827 podStartE2EDuration="4.295075961s" podCreationTimestamp="2026-05-20 15:34:44 +0000 UTC" firstStartedPulling="2026-05-20 15:34:45.2282638 +0000 UTC m=+194.167105335" lastFinishedPulling="2026-05-20 15:34:46.421973919 +0000 UTC m=+195.360815469" observedRunningTime="2026-05-20 15:34:48.29385246 +0000 UTC m=+197.232694016" watchObservedRunningTime="2026-05-20 15:34:48.295075961 +0000 UTC m=+197.233917518"
May 20 15:34:48.319950 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:48.319904    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-8k4jn" podStartSLOduration=3.009012816 podStartE2EDuration="4.319889689s" podCreationTimestamp="2026-05-20 15:34:44 +0000 UTC" firstStartedPulling="2026-05-20 15:34:46.144098755 +0000 UTC m=+195.082940293" lastFinishedPulling="2026-05-20 15:34:47.454975628 +0000 UTC m=+196.393817166" observedRunningTime="2026-05-20 15:34:48.318281097 +0000 UTC m=+197.257122653" watchObservedRunningTime="2026-05-20 15:34:48.319889689 +0000 UTC m=+197.258731299"
May 20 15:34:48.360524 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:48.360468    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-nfcxl" podStartSLOduration=2.907854724 podStartE2EDuration="4.360450709s" podCreationTimestamp="2026-05-20 15:34:44 +0000 UTC" firstStartedPulling="2026-05-20 15:34:46.004459375 +0000 UTC m=+194.943300920" lastFinishedPulling="2026-05-20 15:34:47.457055357 +0000 UTC m=+196.395896905" observedRunningTime="2026-05-20 15:34:48.359045586 +0000 UTC m=+197.297887144" watchObservedRunningTime="2026-05-20 15:34:48.360450709 +0000 UTC m=+197.299292309"
May 20 15:34:58.363885 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:58.363841    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-666497c696-xxwnj"]
May 20 15:34:58.364412 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:34:58.364142    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-666497c696-xxwnj" podUID="c010265c-2cd5-4e4f-9fd5-f75755dc1419"
May 20 15:34:59.308692 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.308656    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:34:59.313322 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.313300    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:34:59.410227 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.410192    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c010265c-2cd5-4e4f-9fd5-f75755dc1419-trusted-ca\") pod \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") "
May 20 15:34:59.410691 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.410242    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-certificates\") pod \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") "
May 20 15:34:59.410691 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.410265    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-bound-sa-token\") pod \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") "
May 20 15:34:59.410691 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.410291    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c010265c-2cd5-4e4f-9fd5-f75755dc1419-ca-trust-extracted\") pod \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") "
May 20 15:34:59.410691 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.410344    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c010265c-2cd5-4e4f-9fd5-f75755dc1419-image-registry-private-configuration\") pod \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") "
May 20 15:34:59.410691 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.410364    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c010265c-2cd5-4e4f-9fd5-f75755dc1419-installation-pull-secrets\") pod \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") "
May 20 15:34:59.410691 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.410515    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8cbs\" (UniqueName: \"kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-kube-api-access-h8cbs\") pod \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\" (UID: \"c010265c-2cd5-4e4f-9fd5-f75755dc1419\") "
May 20 15:34:59.410691 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.410601    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c010265c-2cd5-4e4f-9fd5-f75755dc1419" (UID: "c010265c-2cd5-4e4f-9fd5-f75755dc1419"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:34:59.410691 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.410644    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c010265c-2cd5-4e4f-9fd5-f75755dc1419-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c010265c-2cd5-4e4f-9fd5-f75755dc1419" (UID: "c010265c-2cd5-4e4f-9fd5-f75755dc1419"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:34:59.411304 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.410773    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c010265c-2cd5-4e4f-9fd5-f75755dc1419-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c010265c-2cd5-4e4f-9fd5-f75755dc1419" (UID: "c010265c-2cd5-4e4f-9fd5-f75755dc1419"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:34:59.411304 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.410913    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c010265c-2cd5-4e4f-9fd5-f75755dc1419-trusted-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:34:59.411304 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.410925    2568 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-certificates\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:34:59.411304 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.410938    2568 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c010265c-2cd5-4e4f-9fd5-f75755dc1419-ca-trust-extracted\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:34:59.412913 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.412881    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-kube-api-access-h8cbs" (OuterVolumeSpecName: "kube-api-access-h8cbs") pod "c010265c-2cd5-4e4f-9fd5-f75755dc1419" (UID: "c010265c-2cd5-4e4f-9fd5-f75755dc1419"). InnerVolumeSpecName "kube-api-access-h8cbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:34:59.413061 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.412947    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c010265c-2cd5-4e4f-9fd5-f75755dc1419" (UID: "c010265c-2cd5-4e4f-9fd5-f75755dc1419"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:34:59.413139 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.413106    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c010265c-2cd5-4e4f-9fd5-f75755dc1419-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c010265c-2cd5-4e4f-9fd5-f75755dc1419" (UID: "c010265c-2cd5-4e4f-9fd5-f75755dc1419"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:34:59.413180 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.413161    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c010265c-2cd5-4e4f-9fd5-f75755dc1419-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c010265c-2cd5-4e4f-9fd5-f75755dc1419" (UID: "c010265c-2cd5-4e4f-9fd5-f75755dc1419"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:34:59.512227 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.512189    2568 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-bound-sa-token\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:34:59.512227 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.512217    2568 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c010265c-2cd5-4e4f-9fd5-f75755dc1419-image-registry-private-configuration\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:34:59.512227 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.512229    2568 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c010265c-2cd5-4e4f-9fd5-f75755dc1419-installation-pull-secrets\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:34:59.512227 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:34:59.512239    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8cbs\" (UniqueName: \"kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-kube-api-access-h8cbs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:35:00.311591 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:00.311560    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666497c696-xxwnj"
May 20 15:35:00.346818 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:00.346787    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-666497c696-xxwnj"]
May 20 15:35:00.352201 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:00.352164    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-666497c696-xxwnj"]
May 20 15:35:00.520902 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:00.520862    2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c010265c-2cd5-4e4f-9fd5-f75755dc1419-registry-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:35:01.643454 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:01.643419    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c010265c-2cd5-4e4f-9fd5-f75755dc1419" path="/var/lib/kubelet/pods/c010265c-2cd5-4e4f-9fd5-f75755dc1419/volumes"
May 20 15:35:09.685354 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.685320    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-c55f4dfc7-dhndt"]
May 20 15:35:09.701406 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.701384    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c55f4dfc7-dhndt"]
May 20 15:35:09.701530 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.701486    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:09.703901 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.703877    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 20 15:35:09.704735 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.704715    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 20 15:35:09.704876 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.704856    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 20 15:35:09.704968 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.704743    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 20 15:35:09.704968 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.704764    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8x4xv\""
May 20 15:35:09.705116 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.704779    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 20 15:35:09.705116 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.704785    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 20 15:35:09.705116 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.704845    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 20 15:35:09.802653 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.802631    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-console-serving-cert\") pod \"console-c55f4dfc7-dhndt\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") " pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:09.802782 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.802682    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-console-oauth-config\") pod \"console-c55f4dfc7-dhndt\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") " pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:09.802782 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.802697    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-console-config\") pod \"console-c55f4dfc7-dhndt\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") " pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:09.802869 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.802788    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-oauth-serving-cert\") pod \"console-c55f4dfc7-dhndt\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") " pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:09.802869 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.802817    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-service-ca\") pod \"console-c55f4dfc7-dhndt\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") " pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:09.802869 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.802843    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c9t5\" (UniqueName: \"kubernetes.io/projected/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-kube-api-access-6c9t5\") pod \"console-c55f4dfc7-dhndt\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") " pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:09.903994 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.903969    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-console-oauth-config\") pod \"console-c55f4dfc7-dhndt\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") " pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:09.904126 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.903995    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-console-config\") pod \"console-c55f4dfc7-dhndt\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") " pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:09.904126 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.904056    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-oauth-serving-cert\") pod \"console-c55f4dfc7-dhndt\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") " pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:09.904126 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.904073    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-service-ca\") pod \"console-c55f4dfc7-dhndt\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") " pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:09.904126 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.904093    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6c9t5\" (UniqueName: \"kubernetes.io/projected/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-kube-api-access-6c9t5\") pod \"console-c55f4dfc7-dhndt\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") " pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:09.904126 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.904121    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-console-serving-cert\") pod \"console-c55f4dfc7-dhndt\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") " pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:09.904918 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.904889    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-service-ca\") pod \"console-c55f4dfc7-dhndt\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") " pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:09.904997 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.904891    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-oauth-serving-cert\") pod \"console-c55f4dfc7-dhndt\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") " pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:09.904997 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.904889    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-console-config\") pod \"console-c55f4dfc7-dhndt\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") " pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:09.906541 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.906511    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-console-serving-cert\") pod \"console-c55f4dfc7-dhndt\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") " pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:09.906541 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.906531    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-console-oauth-config\") pod \"console-c55f4dfc7-dhndt\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") " pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:09.913615 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:09.913594    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c9t5\" (UniqueName: \"kubernetes.io/projected/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-kube-api-access-6c9t5\") pod \"console-c55f4dfc7-dhndt\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") " pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:10.010940 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:10.010880    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:10.139979 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:10.139954    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c55f4dfc7-dhndt"]
May 20 15:35:10.142180 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:35:10.142141    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d9e77b_bbfd_4f78_a537_2d30b59e63fc.slice/crio-f6eadf102c565a89cf6161aa6109d73af61d584217b8f4887c10a19a91678e34 WatchSource:0}: Error finding container f6eadf102c565a89cf6161aa6109d73af61d584217b8f4887c10a19a91678e34: Status 404 returned error can't find the container with id f6eadf102c565a89cf6161aa6109d73af61d584217b8f4887c10a19a91678e34
May 20 15:35:10.340232 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:10.340198    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c55f4dfc7-dhndt" event={"ID":"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc","Type":"ContainerStarted","Data":"f6eadf102c565a89cf6161aa6109d73af61d584217b8f4887c10a19a91678e34"}
May 20 15:35:12.348219 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:12.348179    2568 generic.go:358] "Generic (PLEG): container finished" podID="0cbb5691-4ce1-46dd-8a53-d94b5c19edcf" containerID="2da0fd8cf93192cd243883f2fc3e1c2ef041ce83bb910ec6383d76b6da43ae66" exitCode=0
May 20 15:35:12.348655 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:12.348243    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-8ltwr" event={"ID":"0cbb5691-4ce1-46dd-8a53-d94b5c19edcf","Type":"ContainerDied","Data":"2da0fd8cf93192cd243883f2fc3e1c2ef041ce83bb910ec6383d76b6da43ae66"}
May 20 15:35:12.348655 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:12.348607    2568 scope.go:117] "RemoveContainer" containerID="2da0fd8cf93192cd243883f2fc3e1c2ef041ce83bb910ec6383d76b6da43ae66"
May 20 15:35:13.353082 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:13.353038    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-8ltwr" event={"ID":"0cbb5691-4ce1-46dd-8a53-d94b5c19edcf","Type":"ContainerStarted","Data":"62c15a7740f13e14530d5dd39a2d4112807b3b43f677a156c95fd70928f019d4"}
May 20 15:35:13.354379 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:13.354356    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c55f4dfc7-dhndt" event={"ID":"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc","Type":"ContainerStarted","Data":"4126e140f84aac7f9cbc7ddb18709bd86be4411fdb43ba8da4265c37ff261fbd"}
May 20 15:35:13.390493 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:13.390435    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-c55f4dfc7-dhndt" podStartSLOduration=1.466863358 podStartE2EDuration="4.390418922s" podCreationTimestamp="2026-05-20 15:35:09 +0000 UTC" firstStartedPulling="2026-05-20 15:35:10.144005064 +0000 UTC m=+219.082846600" lastFinishedPulling="2026-05-20 15:35:13.067560627 +0000 UTC m=+222.006402164" observedRunningTime="2026-05-20 15:35:13.389109237 +0000 UTC m=+222.327950796" watchObservedRunningTime="2026-05-20 15:35:13.390418922 +0000 UTC m=+222.329260480"
May 20 15:35:19.262885 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.262851    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-78bb7db74d-8ghhg"]
May 20 15:35:19.264904 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.264886    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.271615 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.271590    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 20 15:35:19.277784 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.277758    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78bb7db74d-8ghhg"]
May 20 15:35:19.381203 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.381176    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k2f4\" (UniqueName: \"kubernetes.io/projected/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-kube-api-access-6k2f4\") pod \"console-78bb7db74d-8ghhg\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") " pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.381335 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.381206    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-service-ca\") pod \"console-78bb7db74d-8ghhg\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") " pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.381335 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.381241    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-console-config\") pod \"console-78bb7db74d-8ghhg\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") " pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.381335 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.381261    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-trusted-ca-bundle\") pod \"console-78bb7db74d-8ghhg\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") " pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.381335 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.381283    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-console-serving-cert\") pod \"console-78bb7db74d-8ghhg\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") " pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.381335 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.381321    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-console-oauth-config\") pod \"console-78bb7db74d-8ghhg\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") " pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.381489 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.381371    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-oauth-serving-cert\") pod \"console-78bb7db74d-8ghhg\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") " pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.482002 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.481975    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6k2f4\" (UniqueName: \"kubernetes.io/projected/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-kube-api-access-6k2f4\") pod \"console-78bb7db74d-8ghhg\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") " pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.482133 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.482007    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-service-ca\") pod \"console-78bb7db74d-8ghhg\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") " pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.482133 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.482066    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-console-config\") pod \"console-78bb7db74d-8ghhg\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") " pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.482133 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.482094    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-trusted-ca-bundle\") pod \"console-78bb7db74d-8ghhg\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") " pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.482133 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.482122    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-console-serving-cert\") pod \"console-78bb7db74d-8ghhg\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") " pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.482303 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.482282    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-console-oauth-config\") pod \"console-78bb7db74d-8ghhg\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") " pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.482358 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.482327    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-oauth-serving-cert\") pod \"console-78bb7db74d-8ghhg\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") " pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.482888 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.482811    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-service-ca\") pod \"console-78bb7db74d-8ghhg\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") " pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.482888 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.482876    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-console-config\") pod \"console-78bb7db74d-8ghhg\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") " pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.483194 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.482992    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-trusted-ca-bundle\") pod \"console-78bb7db74d-8ghhg\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") " pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.483194 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.483052    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-oauth-serving-cert\") pod \"console-78bb7db74d-8ghhg\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") " pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.484726 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.484709    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-console-serving-cert\") pod \"console-78bb7db74d-8ghhg\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") " pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.484812 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.484725    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-console-oauth-config\") pod \"console-78bb7db74d-8ghhg\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") " pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.490749 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.490729    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k2f4\" (UniqueName: \"kubernetes.io/projected/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-kube-api-access-6k2f4\") pod \"console-78bb7db74d-8ghhg\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") " pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.574737 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.574711    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:19.704656 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:19.704633    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78bb7db74d-8ghhg"]
May 20 15:35:19.707078 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:35:19.707052    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fdc4fb_3077_40ef_a91b_bfef24c86e9d.slice/crio-a89b4ff22e27b0ae9ec8931a0eb8c0c7945e6c78a90a24c396cf324074e07a0e WatchSource:0}: Error finding container a89b4ff22e27b0ae9ec8931a0eb8c0c7945e6c78a90a24c396cf324074e07a0e: Status 404 returned error can't find the container with id a89b4ff22e27b0ae9ec8931a0eb8c0c7945e6c78a90a24c396cf324074e07a0e
May 20 15:35:20.011305 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:20.011230    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:20.011305 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:20.011267    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:20.015757 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:20.015738    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:20.375354 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:20.375319    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bb7db74d-8ghhg" event={"ID":"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d","Type":"ContainerStarted","Data":"ba3d4f94e0861ecb50393cc2c82a9fb9550413c1306495132a7263f1aa9fabd7"}
May 20 15:35:20.375727 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:20.375358    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bb7db74d-8ghhg" event={"ID":"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d","Type":"ContainerStarted","Data":"a89b4ff22e27b0ae9ec8931a0eb8c0c7945e6c78a90a24c396cf324074e07a0e"}
May 20 15:35:20.379659 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:20.379636    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:20.394489 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:20.394450    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-78bb7db74d-8ghhg" podStartSLOduration=1.394437502 podStartE2EDuration="1.394437502s" podCreationTimestamp="2026-05-20 15:35:19 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:35:20.393718846 +0000 UTC m=+229.332560404" watchObservedRunningTime="2026-05-20 15:35:20.394437502 +0000 UTC m=+229.333279059"
May 20 15:35:28.401398 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:28.401362    2568 generic.go:358] "Generic (PLEG): container finished" podID="1bd01bf5-75e6-45d5-9bf3-0b480b5457e8" containerID="8656468435c128b20865d3316295fc289ca691693a1a231ed4b22fe11f477fb0" exitCode=0
May 20 15:35:28.401763 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:28.401440    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mbpgr" event={"ID":"1bd01bf5-75e6-45d5-9bf3-0b480b5457e8","Type":"ContainerDied","Data":"8656468435c128b20865d3316295fc289ca691693a1a231ed4b22fe11f477fb0"}
May 20 15:35:28.401763 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:28.401756    2568 scope.go:117] "RemoveContainer" containerID="8656468435c128b20865d3316295fc289ca691693a1a231ed4b22fe11f477fb0"
May 20 15:35:29.405630 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:29.405599    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mbpgr" event={"ID":"1bd01bf5-75e6-45d5-9bf3-0b480b5457e8","Type":"ContainerStarted","Data":"f34c5fa1eade1f9cf7e59156d31d67597e90b097353c06239affd40d8b534a3a"}
May 20 15:35:29.575676 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:29.575647    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:29.575830 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:29.575685    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:29.580160 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:29.580139    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:30.413212 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:30.413184    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:35:30.473587 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:30.473548    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c55f4dfc7-dhndt"]
May 20 15:35:43.589969 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:43.589922    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs\") pod \"network-metrics-daemon-skk8q\" (UID: \"3022a27a-6112-46ee-a54b-c5af5bee2a1f\") " pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:35:43.592352 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:43.592330    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3022a27a-6112-46ee-a54b-c5af5bee2a1f-metrics-certs\") pod \"network-metrics-daemon-skk8q\" (UID: \"3022a27a-6112-46ee-a54b-c5af5bee2a1f\") " pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:35:43.643396 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:43.643362    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sz56g\""
May 20 15:35:43.651339 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:43.651323    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skk8q"
May 20 15:35:43.781245 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:43.781127    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-skk8q"]
May 20 15:35:43.784093 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:35:43.784062    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3022a27a_6112_46ee_a54b_c5af5bee2a1f.slice/crio-1009e936e7658851ecd09460bf43283bb154223b7a19df07c0b04fb5edee7281 WatchSource:0}: Error finding container 1009e936e7658851ecd09460bf43283bb154223b7a19df07c0b04fb5edee7281: Status 404 returned error can't find the container with id 1009e936e7658851ecd09460bf43283bb154223b7a19df07c0b04fb5edee7281
May 20 15:35:44.447270 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:44.447241    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-skk8q" event={"ID":"3022a27a-6112-46ee-a54b-c5af5bee2a1f","Type":"ContainerStarted","Data":"1009e936e7658851ecd09460bf43283bb154223b7a19df07c0b04fb5edee7281"}
May 20 15:35:45.451207 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:45.451168    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-skk8q" event={"ID":"3022a27a-6112-46ee-a54b-c5af5bee2a1f","Type":"ContainerStarted","Data":"051dec2a0efb2723daa06acd06525ad50ef0b0e80dcfdbb904ff3a878826a3c2"}
May 20 15:35:45.451207 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:45.451203    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-skk8q" event={"ID":"3022a27a-6112-46ee-a54b-c5af5bee2a1f","Type":"ContainerStarted","Data":"06eca5e5553549fa685dd456e772265e4a0e131ade0667c2badbc5706ca92e0b"}
May 20 15:35:45.472631 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:45.472577    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-skk8q" podStartSLOduration=253.521394268 podStartE2EDuration="4m14.472563635s" podCreationTimestamp="2026-05-20 15:31:31 +0000 UTC" firstStartedPulling="2026-05-20 15:35:43.786053408 +0000 UTC m=+252.724894957" lastFinishedPulling="2026-05-20 15:35:44.737222786 +0000 UTC m=+253.676064324" observedRunningTime="2026-05-20 15:35:45.470635878 +0000 UTC m=+254.409477437" watchObservedRunningTime="2026-05-20 15:35:45.472563635 +0000 UTC m=+254.411405191"
May 20 15:35:46.455186 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:46.455150    2568 generic.go:358] "Generic (PLEG): container finished" podID="afbc552d-4335-4a1c-a5e9-df2444a56e1d" containerID="889d3e4b979520d1895b44b75f0eab0b4e785ef33f1a9a7b273ac1fb882b0174" exitCode=0
May 20 15:35:46.455637 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:46.455230    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-smrgr" event={"ID":"afbc552d-4335-4a1c-a5e9-df2444a56e1d","Type":"ContainerDied","Data":"889d3e4b979520d1895b44b75f0eab0b4e785ef33f1a9a7b273ac1fb882b0174"}
May 20 15:35:46.455637 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:46.455558    2568 scope.go:117] "RemoveContainer" containerID="889d3e4b979520d1895b44b75f0eab0b4e785ef33f1a9a7b273ac1fb882b0174"
May 20 15:35:47.459191 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:47.459157    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-smrgr" event={"ID":"afbc552d-4335-4a1c-a5e9-df2444a56e1d","Type":"ContainerStarted","Data":"00f12e44a8cf629f0b19e9388723a8127f35f0e9a6ff9e126f262182de4e7782"}
May 20 15:35:55.498181 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:55.498098    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-c55f4dfc7-dhndt" podUID="d9d9e77b-bbfd-4f78-a537-2d30b59e63fc" containerName="console" containerID="cri-o://4126e140f84aac7f9cbc7ddb18709bd86be4411fdb43ba8da4265c37ff261fbd" gracePeriod=15
May 20 15:35:55.736606 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:55.736581    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-c55f4dfc7-dhndt_d9d9e77b-bbfd-4f78-a537-2d30b59e63fc/console/0.log"
May 20 15:35:55.736719 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:55.736638    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:55.795966 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:55.795872    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c9t5\" (UniqueName: \"kubernetes.io/projected/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-kube-api-access-6c9t5\") pod \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") "
May 20 15:35:55.795966 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:55.795937    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-console-config\") pod \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") "
May 20 15:35:55.795966 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:55.795968    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-oauth-serving-cert\") pod \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") "
May 20 15:35:55.796265 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:55.795995    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-console-serving-cert\") pod \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") "
May 20 15:35:55.796265 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:55.796075    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-console-oauth-config\") pod \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") "
May 20 15:35:55.796265 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:55.796099    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-service-ca\") pod \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\" (UID: \"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc\") "
May 20 15:35:55.796479 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:55.796446    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d9d9e77b-bbfd-4f78-a537-2d30b59e63fc" (UID: "d9d9e77b-bbfd-4f78-a537-2d30b59e63fc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:35:55.796606 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:55.796450    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-console-config" (OuterVolumeSpecName: "console-config") pod "d9d9e77b-bbfd-4f78-a537-2d30b59e63fc" (UID: "d9d9e77b-bbfd-4f78-a537-2d30b59e63fc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:35:55.796606 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:55.796544    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-service-ca" (OuterVolumeSpecName: "service-ca") pod "d9d9e77b-bbfd-4f78-a537-2d30b59e63fc" (UID: "d9d9e77b-bbfd-4f78-a537-2d30b59e63fc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:35:55.798513 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:55.798483    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d9d9e77b-bbfd-4f78-a537-2d30b59e63fc" (UID: "d9d9e77b-bbfd-4f78-a537-2d30b59e63fc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:35:55.798513 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:55.798501    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d9d9e77b-bbfd-4f78-a537-2d30b59e63fc" (UID: "d9d9e77b-bbfd-4f78-a537-2d30b59e63fc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:35:55.798513 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:55.798505    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-kube-api-access-6c9t5" (OuterVolumeSpecName: "kube-api-access-6c9t5") pod "d9d9e77b-bbfd-4f78-a537-2d30b59e63fc" (UID: "d9d9e77b-bbfd-4f78-a537-2d30b59e63fc"). InnerVolumeSpecName "kube-api-access-6c9t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:35:55.897587 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:55.897549    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-oauth-serving-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:35:55.897587 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:55.897578    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-console-serving-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:35:55.897587 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:55.897589    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-console-oauth-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:35:55.897587 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:55.897601    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-service-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:35:55.897857 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:55.897610    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c9t5\" (UniqueName: \"kubernetes.io/projected/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-kube-api-access-6c9t5\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:35:55.897857 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:55.897619    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc-console-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:35:56.485573 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:56.485547    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-c55f4dfc7-dhndt_d9d9e77b-bbfd-4f78-a537-2d30b59e63fc/console/0.log"
May 20 15:35:56.485744 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:56.485589    2568 generic.go:358] "Generic (PLEG): container finished" podID="d9d9e77b-bbfd-4f78-a537-2d30b59e63fc" containerID="4126e140f84aac7f9cbc7ddb18709bd86be4411fdb43ba8da4265c37ff261fbd" exitCode=2
May 20 15:35:56.485744 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:56.485634    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c55f4dfc7-dhndt" event={"ID":"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc","Type":"ContainerDied","Data":"4126e140f84aac7f9cbc7ddb18709bd86be4411fdb43ba8da4265c37ff261fbd"}
May 20 15:35:56.485744 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:56.485657    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c55f4dfc7-dhndt" event={"ID":"d9d9e77b-bbfd-4f78-a537-2d30b59e63fc","Type":"ContainerDied","Data":"f6eadf102c565a89cf6161aa6109d73af61d584217b8f4887c10a19a91678e34"}
May 20 15:35:56.485744 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:56.485662    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c55f4dfc7-dhndt"
May 20 15:35:56.485744 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:56.485671    2568 scope.go:117] "RemoveContainer" containerID="4126e140f84aac7f9cbc7ddb18709bd86be4411fdb43ba8da4265c37ff261fbd"
May 20 15:35:56.495242 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:56.495219    2568 scope.go:117] "RemoveContainer" containerID="4126e140f84aac7f9cbc7ddb18709bd86be4411fdb43ba8da4265c37ff261fbd"
May 20 15:35:56.495505 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:35:56.495487    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4126e140f84aac7f9cbc7ddb18709bd86be4411fdb43ba8da4265c37ff261fbd\": container with ID starting with 4126e140f84aac7f9cbc7ddb18709bd86be4411fdb43ba8da4265c37ff261fbd not found: ID does not exist" containerID="4126e140f84aac7f9cbc7ddb18709bd86be4411fdb43ba8da4265c37ff261fbd"
May 20 15:35:56.495564 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:56.495515    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4126e140f84aac7f9cbc7ddb18709bd86be4411fdb43ba8da4265c37ff261fbd"} err="failed to get container status \"4126e140f84aac7f9cbc7ddb18709bd86be4411fdb43ba8da4265c37ff261fbd\": rpc error: code = NotFound desc = could not find container \"4126e140f84aac7f9cbc7ddb18709bd86be4411fdb43ba8da4265c37ff261fbd\": container with ID starting with 4126e140f84aac7f9cbc7ddb18709bd86be4411fdb43ba8da4265c37ff261fbd not found: ID does not exist"
May 20 15:35:56.506650 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:56.506625    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c55f4dfc7-dhndt"]
May 20 15:35:56.510306 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:56.510283    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c55f4dfc7-dhndt"]
May 20 15:35:57.642821 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:35:57.642784    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9d9e77b-bbfd-4f78-a537-2d30b59e63fc" path="/var/lib/kubelet/pods/d9d9e77b-bbfd-4f78-a537-2d30b59e63fc/volumes"
May 20 15:36:12.131785 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:36:12.131738    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-w8bsv" podUID="9bcdce67-2f01-4c5e-a20f-c976159f8683"
May 20 15:36:12.131785 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:36:12.131753    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj" podUID="94373970-3fe5-4bbf-a05a-f4913b27e8d0"
May 20 15:36:12.131785 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:36:12.131754    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-szz5l" podUID="fbac1f01-44d4-4042-a375-5a6a14f94d47"
May 20 15:36:12.534238 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:12.534165    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-szz5l"
May 20 15:36:12.534372 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:12.534273    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w8bsv"
May 20 15:36:12.534466 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:12.534449    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj"
May 20 15:36:13.346946 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.346908    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f8d856c4d-wpxn8"]
May 20 15:36:13.347348 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.347304    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9d9e77b-bbfd-4f78-a537-2d30b59e63fc" containerName="console"
May 20 15:36:13.347348 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.347317    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9d9e77b-bbfd-4f78-a537-2d30b59e63fc" containerName="console"
May 20 15:36:13.347419 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.347373    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="d9d9e77b-bbfd-4f78-a537-2d30b59e63fc" containerName="console"
May 20 15:36:13.349982 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.349966    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.371061 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.371000    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f8d856c4d-wpxn8"]
May 20 15:36:13.449995 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.449961    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-service-ca\") pod \"console-6f8d856c4d-wpxn8\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") " pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.450249 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.449997    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-trusted-ca-bundle\") pod \"console-6f8d856c4d-wpxn8\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") " pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.450249 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.450047    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae57088e-8522-4f76-91d3-06abe21d5577-console-oauth-config\") pod \"console-6f8d856c4d-wpxn8\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") " pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.450249 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.450189    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-console-config\") pod \"console-6f8d856c4d-wpxn8\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") " pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.450249 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.450234    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp9x5\" (UniqueName: \"kubernetes.io/projected/ae57088e-8522-4f76-91d3-06abe21d5577-kube-api-access-hp9x5\") pod \"console-6f8d856c4d-wpxn8\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") " pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.450505 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.450313    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-oauth-serving-cert\") pod \"console-6f8d856c4d-wpxn8\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") " pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.450505 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.450346    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae57088e-8522-4f76-91d3-06abe21d5577-console-serving-cert\") pod \"console-6f8d856c4d-wpxn8\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") " pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.551572 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.551531    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hp9x5\" (UniqueName: \"kubernetes.io/projected/ae57088e-8522-4f76-91d3-06abe21d5577-kube-api-access-hp9x5\") pod \"console-6f8d856c4d-wpxn8\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") " pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.551767 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.551604    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-oauth-serving-cert\") pod \"console-6f8d856c4d-wpxn8\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") " pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.551767 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.551634    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae57088e-8522-4f76-91d3-06abe21d5577-console-serving-cert\") pod \"console-6f8d856c4d-wpxn8\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") " pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.551767 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.551694    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-service-ca\") pod \"console-6f8d856c4d-wpxn8\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") " pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.551767 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.551719    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-trusted-ca-bundle\") pod \"console-6f8d856c4d-wpxn8\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") " pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.551767 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.551744    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae57088e-8522-4f76-91d3-06abe21d5577-console-oauth-config\") pod \"console-6f8d856c4d-wpxn8\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") " pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.552140 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.551923    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-console-config\") pod \"console-6f8d856c4d-wpxn8\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") " pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.552545 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.552467    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-service-ca\") pod \"console-6f8d856c4d-wpxn8\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") " pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.552545 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.552467    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-oauth-serving-cert\") pod \"console-6f8d856c4d-wpxn8\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") " pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.552735 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.552711    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-trusted-ca-bundle\") pod \"console-6f8d856c4d-wpxn8\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") " pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.552796 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.552775    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-console-config\") pod \"console-6f8d856c4d-wpxn8\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") " pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.554420 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.554401    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae57088e-8522-4f76-91d3-06abe21d5577-console-oauth-config\") pod \"console-6f8d856c4d-wpxn8\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") " pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.554742 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.554723    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae57088e-8522-4f76-91d3-06abe21d5577-console-serving-cert\") pod \"console-6f8d856c4d-wpxn8\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") " pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.559997 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.559972    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp9x5\" (UniqueName: \"kubernetes.io/projected/ae57088e-8522-4f76-91d3-06abe21d5577-kube-api-access-hp9x5\") pod \"console-6f8d856c4d-wpxn8\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") " pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.658559 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.658473    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:13.790448 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:13.790424    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f8d856c4d-wpxn8"]
May 20 15:36:13.792922 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:36:13.792893    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae57088e_8522_4f76_91d3_06abe21d5577.slice/crio-b4f2e51b9b15ce1a79367796b2dbc22302135150d6b4260daf7e3a63b9234162 WatchSource:0}: Error finding container b4f2e51b9b15ce1a79367796b2dbc22302135150d6b4260daf7e3a63b9234162: Status 404 returned error can't find the container with id b4f2e51b9b15ce1a79367796b2dbc22302135150d6b4260daf7e3a63b9234162
May 20 15:36:14.540784 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:14.540746    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f8d856c4d-wpxn8" event={"ID":"ae57088e-8522-4f76-91d3-06abe21d5577","Type":"ContainerStarted","Data":"1a4e190b20748453178e8683c70cb0e4d0173ed07320e317974a40175524b5e5"}
May 20 15:36:14.540784 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:14.540786    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f8d856c4d-wpxn8" event={"ID":"ae57088e-8522-4f76-91d3-06abe21d5577","Type":"ContainerStarted","Data":"b4f2e51b9b15ce1a79367796b2dbc22302135150d6b4260daf7e3a63b9234162"}
May 20 15:36:14.561111 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:14.561053    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-6f8d856c4d-wpxn8" podStartSLOduration=1.561014133 podStartE2EDuration="1.561014133s" podCreationTimestamp="2026-05-20 15:36:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:36:14.559063995 +0000 UTC m=+283.497905549" watchObservedRunningTime="2026-05-20 15:36:14.561014133 +0000 UTC m=+283.499855691"
May 20 15:36:15.974812 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:15.974762    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-2fdvj\" (UID: \"94373970-3fe5-4bbf-a05a-f4913b27e8d0\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj"
May 20 15:36:15.977312 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:15.977280    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/94373970-3fe5-4bbf-a05a-f4913b27e8d0-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-2fdvj\" (UID: \"94373970-3fe5-4bbf-a05a-f4913b27e8d0\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj"
May 20 15:36:16.075344 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:16.075313    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert\") pod \"ingress-canary-szz5l\" (UID: \"fbac1f01-44d4-4042-a375-5a6a14f94d47\") " pod="openshift-ingress-canary/ingress-canary-szz5l"
May 20 15:36:16.075493 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:16.075362    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls\") pod \"dns-default-w8bsv\" (UID: \"9bcdce67-2f01-4c5e-a20f-c976159f8683\") " pod="openshift-dns/dns-default-w8bsv"
May 20 15:36:16.077743 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:16.077714    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bcdce67-2f01-4c5e-a20f-c976159f8683-metrics-tls\") pod \"dns-default-w8bsv\" (UID: \"9bcdce67-2f01-4c5e-a20f-c976159f8683\") " pod="openshift-dns/dns-default-w8bsv"
May 20 15:36:16.078058 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:16.078042    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbac1f01-44d4-4042-a375-5a6a14f94d47-cert\") pod \"ingress-canary-szz5l\" (UID: \"fbac1f01-44d4-4042-a375-5a6a14f94d47\") " pod="openshift-ingress-canary/ingress-canary-szz5l"
May 20 15:36:16.138166 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:16.138140    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-trd7m\""
May 20 15:36:16.138311 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:16.138140    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ffrjz\""
May 20 15:36:16.138311 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:16.138185    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4srs2\""
May 20 15:36:16.145943 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:16.145917    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj"
May 20 15:36:16.146091 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:16.145950    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w8bsv"
May 20 15:36:16.146091 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:16.146007    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-szz5l"
May 20 15:36:16.294771 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:16.294742    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj"]
May 20 15:36:16.299428 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:36:16.299371    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94373970_3fe5_4bbf_a05a_f4913b27e8d0.slice/crio-bba529be712e266edf223db316c06ba038f88d7d16dc6fb82c0aba82c0580a9f WatchSource:0}: Error finding container bba529be712e266edf223db316c06ba038f88d7d16dc6fb82c0aba82c0580a9f: Status 404 returned error can't find the container with id bba529be712e266edf223db316c06ba038f88d7d16dc6fb82c0aba82c0580a9f
May 20 15:36:16.523130 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:16.523108    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-szz5l"]
May 20 15:36:16.525368 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:36:16.525337    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbac1f01_44d4_4042_a375_5a6a14f94d47.slice/crio-344efaa9eaebe0b4e36fbab4ff179a66d9acb41ec2c97c3a3788acd7b18b49c0 WatchSource:0}: Error finding container 344efaa9eaebe0b4e36fbab4ff179a66d9acb41ec2c97c3a3788acd7b18b49c0: Status 404 returned error can't find the container with id 344efaa9eaebe0b4e36fbab4ff179a66d9acb41ec2c97c3a3788acd7b18b49c0
May 20 15:36:16.526007 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:16.525990    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w8bsv"]
May 20 15:36:16.528314 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:36:16.528295    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bcdce67_2f01_4c5e_a20f_c976159f8683.slice/crio-b78479c2e703ccbac947aa1f93e7479f3f5a2f04e62e071cbb91077d7d155319 WatchSource:0}: Error finding container b78479c2e703ccbac947aa1f93e7479f3f5a2f04e62e071cbb91077d7d155319: Status 404 returned error can't find the container with id b78479c2e703ccbac947aa1f93e7479f3f5a2f04e62e071cbb91077d7d155319
May 20 15:36:16.551285 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:16.551243    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-szz5l" event={"ID":"fbac1f01-44d4-4042-a375-5a6a14f94d47","Type":"ContainerStarted","Data":"344efaa9eaebe0b4e36fbab4ff179a66d9acb41ec2c97c3a3788acd7b18b49c0"}
May 20 15:36:16.552449 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:16.552427    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w8bsv" event={"ID":"9bcdce67-2f01-4c5e-a20f-c976159f8683","Type":"ContainerStarted","Data":"b78479c2e703ccbac947aa1f93e7479f3f5a2f04e62e071cbb91077d7d155319"}
May 20 15:36:16.553399 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:16.553381    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj" event={"ID":"94373970-3fe5-4bbf-a05a-f4913b27e8d0","Type":"ContainerStarted","Data":"bba529be712e266edf223db316c06ba038f88d7d16dc6fb82c0aba82c0580a9f"}
May 20 15:36:17.558414 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:17.558374    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj" event={"ID":"94373970-3fe5-4bbf-a05a-f4913b27e8d0","Type":"ContainerStarted","Data":"16f8f182682e41474fe02fcf3d83714c44d0e9eed4f548f25b9fee140a12665a"}
May 20 15:36:17.576296 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:17.576210    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-2fdvj" podStartSLOduration=266.5175694 podStartE2EDuration="4m27.57618941s" podCreationTimestamp="2026-05-20 15:31:50 +0000 UTC" firstStartedPulling="2026-05-20 15:36:16.301656636 +0000 UTC m=+285.240498172" lastFinishedPulling="2026-05-20 15:36:17.360276638 +0000 UTC m=+286.299118182" observedRunningTime="2026-05-20 15:36:17.5752611 +0000 UTC m=+286.514102659" watchObservedRunningTime="2026-05-20 15:36:17.57618941 +0000 UTC m=+286.515030968"
May 20 15:36:18.566449 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:18.566395    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-szz5l" event={"ID":"fbac1f01-44d4-4042-a375-5a6a14f94d47","Type":"ContainerStarted","Data":"879cce7a077864c462927d4cb46215dd050e7c6c350616ecb09603955d94e0b8"}
May 20 15:36:18.568121 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:18.568086    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w8bsv" event={"ID":"9bcdce67-2f01-4c5e-a20f-c976159f8683","Type":"ContainerStarted","Data":"da1da128af1065a7ecde24c3ec4a1a160519c6c1c7ab55b0e9dddaf276d5398a"}
May 20 15:36:18.583303 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:18.583252    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-szz5l" podStartSLOduration=251.689487773 podStartE2EDuration="4m13.58323502s" podCreationTimestamp="2026-05-20 15:32:05 +0000 UTC" firstStartedPulling="2026-05-20 15:36:16.52746189 +0000 UTC m=+285.466303424" lastFinishedPulling="2026-05-20 15:36:18.421209136 +0000 UTC m=+287.360050671" observedRunningTime="2026-05-20 15:36:18.582139418 +0000 UTC m=+287.520980975" watchObservedRunningTime="2026-05-20 15:36:18.58323502 +0000 UTC m=+287.522076579"
May 20 15:36:19.573391 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:19.573348    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w8bsv" event={"ID":"9bcdce67-2f01-4c5e-a20f-c976159f8683","Type":"ContainerStarted","Data":"d3485f2232041da3e90f95f4d29a4769f83adedc3026908c6dec71438d4fd605"}
May 20 15:36:19.573794 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:19.573527    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w8bsv"
May 20 15:36:19.592151 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:19.592106    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-w8bsv" podStartSLOduration=252.704793512 podStartE2EDuration="4m14.592092437s" podCreationTimestamp="2026-05-20 15:32:05 +0000 UTC" firstStartedPulling="2026-05-20 15:36:16.530040947 +0000 UTC m=+285.468882482" lastFinishedPulling="2026-05-20 15:36:18.417339869 +0000 UTC m=+287.356181407" observedRunningTime="2026-05-20 15:36:19.590918207 +0000 UTC m=+288.529759767" watchObservedRunningTime="2026-05-20 15:36:19.592092437 +0000 UTC m=+288.530933994"
May 20 15:36:23.658697 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:23.658665    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:23.659086 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:23.658711    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:23.663637 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:23.663614    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:24.594710 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:24.594683    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:36:24.641704 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:24.641673    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78bb7db74d-8ghhg"]
May 20 15:36:29.579902 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:29.579869    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w8bsv"
May 20 15:36:31.564574 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:31.564549    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 15:36:31.565446 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:31.565429    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 15:36:31.572268 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:31.572252    2568 kubelet.go:1628] "Image garbage collection succeeded"
May 20 15:36:49.663855 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:49.663822    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-78bb7db74d-8ghhg" podUID="d4fdc4fb-3077-40ef-a91b-bfef24c86e9d" containerName="console" containerID="cri-o://ba3d4f94e0861ecb50393cc2c82a9fb9550413c1306495132a7263f1aa9fabd7" gracePeriod=15
May 20 15:36:49.915890 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:49.915830    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-78bb7db74d-8ghhg_d4fdc4fb-3077-40ef-a91b-bfef24c86e9d/console/0.log"
May 20 15:36:49.915995 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:49.915893    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:36:49.957498 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:49.957472    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-service-ca\") pod \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") "
May 20 15:36:49.957667 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:49.957507    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-console-serving-cert\") pod \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") "
May 20 15:36:49.957667 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:49.957565    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k2f4\" (UniqueName: \"kubernetes.io/projected/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-kube-api-access-6k2f4\") pod \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") "
May 20 15:36:49.957667 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:49.957592    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-console-config\") pod \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") "
May 20 15:36:49.957667 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:49.957612    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-trusted-ca-bundle\") pod \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") "
May 20 15:36:49.957880 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:49.957701    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-oauth-serving-cert\") pod \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") "
May 20 15:36:49.957880 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:49.957783    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-console-oauth-config\") pod \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\" (UID: \"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d\") "
May 20 15:36:49.957981 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:49.957961    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-service-ca" (OuterVolumeSpecName: "service-ca") pod "d4fdc4fb-3077-40ef-a91b-bfef24c86e9d" (UID: "d4fdc4fb-3077-40ef-a91b-bfef24c86e9d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:36:49.958080 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:49.958053    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d4fdc4fb-3077-40ef-a91b-bfef24c86e9d" (UID: "d4fdc4fb-3077-40ef-a91b-bfef24c86e9d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:36:49.958080 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:49.958061    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-console-config" (OuterVolumeSpecName: "console-config") pod "d4fdc4fb-3077-40ef-a91b-bfef24c86e9d" (UID: "d4fdc4fb-3077-40ef-a91b-bfef24c86e9d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:36:49.958186 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:49.958066    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d4fdc4fb-3077-40ef-a91b-bfef24c86e9d" (UID: "d4fdc4fb-3077-40ef-a91b-bfef24c86e9d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:36:49.958186 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:49.958067    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-service-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:36:49.959881 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:49.959851    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-kube-api-access-6k2f4" (OuterVolumeSpecName: "kube-api-access-6k2f4") pod "d4fdc4fb-3077-40ef-a91b-bfef24c86e9d" (UID: "d4fdc4fb-3077-40ef-a91b-bfef24c86e9d"). InnerVolumeSpecName "kube-api-access-6k2f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:36:49.959881 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:49.959862    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d4fdc4fb-3077-40ef-a91b-bfef24c86e9d" (UID: "d4fdc4fb-3077-40ef-a91b-bfef24c86e9d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:36:49.960049 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:49.959901    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d4fdc4fb-3077-40ef-a91b-bfef24c86e9d" (UID: "d4fdc4fb-3077-40ef-a91b-bfef24c86e9d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:36:50.058760 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:50.058728    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k2f4\" (UniqueName: \"kubernetes.io/projected/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-kube-api-access-6k2f4\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:36:50.058760 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:50.058756    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-console-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:36:50.058760 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:50.058766    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-trusted-ca-bundle\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:36:50.058968 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:50.058774    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-oauth-serving-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:36:50.058968 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:50.058783    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-console-oauth-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:36:50.058968 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:50.058791    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d-console-serving-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:36:50.667963 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:50.667939    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-78bb7db74d-8ghhg_d4fdc4fb-3077-40ef-a91b-bfef24c86e9d/console/0.log"
May 20 15:36:50.668378 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:50.667981    2568 generic.go:358] "Generic (PLEG): container finished" podID="d4fdc4fb-3077-40ef-a91b-bfef24c86e9d" containerID="ba3d4f94e0861ecb50393cc2c82a9fb9550413c1306495132a7263f1aa9fabd7" exitCode=2
May 20 15:36:50.668378 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:50.668049    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bb7db74d-8ghhg"
May 20 15:36:50.668378 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:50.668067    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bb7db74d-8ghhg" event={"ID":"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d","Type":"ContainerDied","Data":"ba3d4f94e0861ecb50393cc2c82a9fb9550413c1306495132a7263f1aa9fabd7"}
May 20 15:36:50.668378 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:50.668099    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bb7db74d-8ghhg" event={"ID":"d4fdc4fb-3077-40ef-a91b-bfef24c86e9d","Type":"ContainerDied","Data":"a89b4ff22e27b0ae9ec8931a0eb8c0c7945e6c78a90a24c396cf324074e07a0e"}
May 20 15:36:50.668378 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:50.668113    2568 scope.go:117] "RemoveContainer" containerID="ba3d4f94e0861ecb50393cc2c82a9fb9550413c1306495132a7263f1aa9fabd7"
May 20 15:36:50.680099 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:50.680080    2568 scope.go:117] "RemoveContainer" containerID="ba3d4f94e0861ecb50393cc2c82a9fb9550413c1306495132a7263f1aa9fabd7"
May 20 15:36:50.680386 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:36:50.680363    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba3d4f94e0861ecb50393cc2c82a9fb9550413c1306495132a7263f1aa9fabd7\": container with ID starting with ba3d4f94e0861ecb50393cc2c82a9fb9550413c1306495132a7263f1aa9fabd7 not found: ID does not exist" containerID="ba3d4f94e0861ecb50393cc2c82a9fb9550413c1306495132a7263f1aa9fabd7"
May 20 15:36:50.680478 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:50.680392    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba3d4f94e0861ecb50393cc2c82a9fb9550413c1306495132a7263f1aa9fabd7"} err="failed to get container status \"ba3d4f94e0861ecb50393cc2c82a9fb9550413c1306495132a7263f1aa9fabd7\": rpc error: code = NotFound desc = could not find container \"ba3d4f94e0861ecb50393cc2c82a9fb9550413c1306495132a7263f1aa9fabd7\": container with ID starting with ba3d4f94e0861ecb50393cc2c82a9fb9550413c1306495132a7263f1aa9fabd7 not found: ID does not exist"
May 20 15:36:50.690098 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:50.690074    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78bb7db74d-8ghhg"]
May 20 15:36:50.696961 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:50.696939    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78bb7db74d-8ghhg"]
May 20 15:36:51.643317 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:36:51.643282    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4fdc4fb-3077-40ef-a91b-bfef24c86e9d" path="/var/lib/kubelet/pods/d4fdc4fb-3077-40ef-a91b-bfef24c86e9d/volumes"
May 20 15:37:46.381781 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.381743    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-58d54fb74b-ldhr7"]
May 20 15:37:46.382249 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.382071    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4fdc4fb-3077-40ef-a91b-bfef24c86e9d" containerName="console"
May 20 15:37:46.382249 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.382083    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4fdc4fb-3077-40ef-a91b-bfef24c86e9d" containerName="console"
May 20 15:37:46.382249 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.382139    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="d4fdc4fb-3077-40ef-a91b-bfef24c86e9d" containerName="console"
May 20 15:37:46.384743 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.384722    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.396347 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.396318    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58d54fb74b-ldhr7"]
May 20 15:37:46.410423 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.410394    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-trusted-ca-bundle\") pod \"console-58d54fb74b-ldhr7\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") " pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.410561 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.410431    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4a13f6a-33c1-481d-a27a-82931348c609-console-serving-cert\") pod \"console-58d54fb74b-ldhr7\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") " pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.410561 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.410467    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-service-ca\") pod \"console-58d54fb74b-ldhr7\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") " pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.410561 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.410536    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-oauth-serving-cert\") pod \"console-58d54fb74b-ldhr7\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") " pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.410674 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.410617    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4a13f6a-33c1-481d-a27a-82931348c609-console-oauth-config\") pod \"console-58d54fb74b-ldhr7\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") " pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.410674 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.410634    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-console-config\") pod \"console-58d54fb74b-ldhr7\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") " pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.410674 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.410654    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j58g2\" (UniqueName: \"kubernetes.io/projected/a4a13f6a-33c1-481d-a27a-82931348c609-kube-api-access-j58g2\") pod \"console-58d54fb74b-ldhr7\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") " pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.511064 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.511009    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4a13f6a-33c1-481d-a27a-82931348c609-console-oauth-config\") pod \"console-58d54fb74b-ldhr7\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") " pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.511244 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.511068    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-console-config\") pod \"console-58d54fb74b-ldhr7\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") " pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.511244 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.511091    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j58g2\" (UniqueName: \"kubernetes.io/projected/a4a13f6a-33c1-481d-a27a-82931348c609-kube-api-access-j58g2\") pod \"console-58d54fb74b-ldhr7\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") " pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.511244 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.511115    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-trusted-ca-bundle\") pod \"console-58d54fb74b-ldhr7\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") " pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.511244 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.511133    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4a13f6a-33c1-481d-a27a-82931348c609-console-serving-cert\") pod \"console-58d54fb74b-ldhr7\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") " pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.511244 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.511158    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-service-ca\") pod \"console-58d54fb74b-ldhr7\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") " pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.511244 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.511174    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-oauth-serving-cert\") pod \"console-58d54fb74b-ldhr7\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") " pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.512008 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.511946    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-service-ca\") pod \"console-58d54fb74b-ldhr7\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") " pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.512008 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.511982    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-oauth-serving-cert\") pod \"console-58d54fb74b-ldhr7\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") " pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.512211 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.512190    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-console-config\") pod \"console-58d54fb74b-ldhr7\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") " pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.512270 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.512222    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-trusted-ca-bundle\") pod \"console-58d54fb74b-ldhr7\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") " pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.513754 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.513713    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4a13f6a-33c1-481d-a27a-82931348c609-console-oauth-config\") pod \"console-58d54fb74b-ldhr7\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") " pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.513853 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.513836    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4a13f6a-33c1-481d-a27a-82931348c609-console-serving-cert\") pod \"console-58d54fb74b-ldhr7\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") " pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.520081 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.520049    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j58g2\" (UniqueName: \"kubernetes.io/projected/a4a13f6a-33c1-481d-a27a-82931348c609-kube-api-access-j58g2\") pod \"console-58d54fb74b-ldhr7\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") " pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.694062 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.693985    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:46.821679 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.821649    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58d54fb74b-ldhr7"]
May 20 15:37:46.824093 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:37:46.824064    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a13f6a_33c1_481d_a27a_82931348c609.slice/crio-54fede5c838ad68d38bdd9924e2d1bdb4baac034a402daef37f0df50f9566743 WatchSource:0}: Error finding container 54fede5c838ad68d38bdd9924e2d1bdb4baac034a402daef37f0df50f9566743: Status 404 returned error can't find the container with id 54fede5c838ad68d38bdd9924e2d1bdb4baac034a402daef37f0df50f9566743
May 20 15:37:46.825909 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.825893    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:37:46.830087 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:46.829936    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d54fb74b-ldhr7" event={"ID":"a4a13f6a-33c1-481d-a27a-82931348c609","Type":"ContainerStarted","Data":"54fede5c838ad68d38bdd9924e2d1bdb4baac034a402daef37f0df50f9566743"}
May 20 15:37:47.834267 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:47.834226    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d54fb74b-ldhr7" event={"ID":"a4a13f6a-33c1-481d-a27a-82931348c609","Type":"ContainerStarted","Data":"a5d58e65cd045a96a967d3598c7e90b3c7e2a6bc76ada96574dd734ae9bce832"}
May 20 15:37:47.860333 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:47.860285    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-58d54fb74b-ldhr7" podStartSLOduration=1.860271634 podStartE2EDuration="1.860271634s" podCreationTimestamp="2026-05-20 15:37:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:37:47.858653815 +0000 UTC m=+376.797495376" watchObservedRunningTime="2026-05-20 15:37:47.860271634 +0000 UTC m=+376.799113190"
May 20 15:37:56.695121 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:56.695054    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:56.695121 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:56.695136    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:56.700807 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:56.700780    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:56.864272 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:56.864243    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:37:56.922808 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:37:56.922778    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f8d856c4d-wpxn8"]
May 20 15:38:21.943711 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:21.943652    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-6f8d856c4d-wpxn8" podUID="ae57088e-8522-4f76-91d3-06abe21d5577" containerName="console" containerID="cri-o://1a4e190b20748453178e8683c70cb0e4d0173ed07320e317974a40175524b5e5" gracePeriod=15
May 20 15:38:22.179715 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.179688    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6f8d856c4d-wpxn8_ae57088e-8522-4f76-91d3-06abe21d5577/console/0.log"
May 20 15:38:22.179845 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.179752    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:38:22.220669 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.220573    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae57088e-8522-4f76-91d3-06abe21d5577-console-oauth-config\") pod \"ae57088e-8522-4f76-91d3-06abe21d5577\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") "
May 20 15:38:22.220669 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.220613    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-oauth-serving-cert\") pod \"ae57088e-8522-4f76-91d3-06abe21d5577\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") "
May 20 15:38:22.220669 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.220631    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-console-config\") pod \"ae57088e-8522-4f76-91d3-06abe21d5577\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") "
May 20 15:38:22.220669 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.220654    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hp9x5\" (UniqueName: \"kubernetes.io/projected/ae57088e-8522-4f76-91d3-06abe21d5577-kube-api-access-hp9x5\") pod \"ae57088e-8522-4f76-91d3-06abe21d5577\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") "
May 20 15:38:22.220993 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.220703    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae57088e-8522-4f76-91d3-06abe21d5577-console-serving-cert\") pod \"ae57088e-8522-4f76-91d3-06abe21d5577\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") "
May 20 15:38:22.220993 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.220741    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-service-ca\") pod \"ae57088e-8522-4f76-91d3-06abe21d5577\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") "
May 20 15:38:22.220993 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.220765    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-trusted-ca-bundle\") pod \"ae57088e-8522-4f76-91d3-06abe21d5577\" (UID: \"ae57088e-8522-4f76-91d3-06abe21d5577\") "
May 20 15:38:22.221198 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.221092    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-console-config" (OuterVolumeSpecName: "console-config") pod "ae57088e-8522-4f76-91d3-06abe21d5577" (UID: "ae57088e-8522-4f76-91d3-06abe21d5577"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:38:22.221284 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.221253    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-service-ca" (OuterVolumeSpecName: "service-ca") pod "ae57088e-8522-4f76-91d3-06abe21d5577" (UID: "ae57088e-8522-4f76-91d3-06abe21d5577"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:38:22.221342 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.221289    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ae57088e-8522-4f76-91d3-06abe21d5577" (UID: "ae57088e-8522-4f76-91d3-06abe21d5577"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:38:22.221400 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.221355    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ae57088e-8522-4f76-91d3-06abe21d5577" (UID: "ae57088e-8522-4f76-91d3-06abe21d5577"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:38:22.223111 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.223086    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae57088e-8522-4f76-91d3-06abe21d5577-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ae57088e-8522-4f76-91d3-06abe21d5577" (UID: "ae57088e-8522-4f76-91d3-06abe21d5577"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:38:22.223221 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.223111    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae57088e-8522-4f76-91d3-06abe21d5577-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ae57088e-8522-4f76-91d3-06abe21d5577" (UID: "ae57088e-8522-4f76-91d3-06abe21d5577"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:38:22.223258 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.223229    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae57088e-8522-4f76-91d3-06abe21d5577-kube-api-access-hp9x5" (OuterVolumeSpecName: "kube-api-access-hp9x5") pod "ae57088e-8522-4f76-91d3-06abe21d5577" (UID: "ae57088e-8522-4f76-91d3-06abe21d5577"). InnerVolumeSpecName "kube-api-access-hp9x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:38:22.321527 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.321491    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae57088e-8522-4f76-91d3-06abe21d5577-console-oauth-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:38:22.321527 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.321524    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-oauth-serving-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:38:22.321527 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.321533    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-console-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:38:22.321763 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.321543    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp9x5\" (UniqueName: \"kubernetes.io/projected/ae57088e-8522-4f76-91d3-06abe21d5577-kube-api-access-hp9x5\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:38:22.321763 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.321554    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae57088e-8522-4f76-91d3-06abe21d5577-console-serving-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:38:22.321763 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.321563    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-service-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:38:22.321763 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.321571    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae57088e-8522-4f76-91d3-06abe21d5577-trusted-ca-bundle\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:38:22.937458 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.937430    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6f8d856c4d-wpxn8_ae57088e-8522-4f76-91d3-06abe21d5577/console/0.log"
May 20 15:38:22.937653 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.937469    2568 generic.go:358] "Generic (PLEG): container finished" podID="ae57088e-8522-4f76-91d3-06abe21d5577" containerID="1a4e190b20748453178e8683c70cb0e4d0173ed07320e317974a40175524b5e5" exitCode=2
May 20 15:38:22.937653 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.937527    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f8d856c4d-wpxn8" event={"ID":"ae57088e-8522-4f76-91d3-06abe21d5577","Type":"ContainerDied","Data":"1a4e190b20748453178e8683c70cb0e4d0173ed07320e317974a40175524b5e5"}
May 20 15:38:22.937653 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.937547    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f8d856c4d-wpxn8" event={"ID":"ae57088e-8522-4f76-91d3-06abe21d5577","Type":"ContainerDied","Data":"b4f2e51b9b15ce1a79367796b2dbc22302135150d6b4260daf7e3a63b9234162"}
May 20 15:38:22.937653 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.937560    2568 scope.go:117] "RemoveContainer" containerID="1a4e190b20748453178e8683c70cb0e4d0173ed07320e317974a40175524b5e5"
May 20 15:38:22.937653 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.937558    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f8d856c4d-wpxn8"
May 20 15:38:22.946357 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.946209    2568 scope.go:117] "RemoveContainer" containerID="1a4e190b20748453178e8683c70cb0e4d0173ed07320e317974a40175524b5e5"
May 20 15:38:22.946584 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:38:22.946467    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a4e190b20748453178e8683c70cb0e4d0173ed07320e317974a40175524b5e5\": container with ID starting with 1a4e190b20748453178e8683c70cb0e4d0173ed07320e317974a40175524b5e5 not found: ID does not exist" containerID="1a4e190b20748453178e8683c70cb0e4d0173ed07320e317974a40175524b5e5"
May 20 15:38:22.946584 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.946492    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4e190b20748453178e8683c70cb0e4d0173ed07320e317974a40175524b5e5"} err="failed to get container status \"1a4e190b20748453178e8683c70cb0e4d0173ed07320e317974a40175524b5e5\": rpc error: code = NotFound desc = could not find container \"1a4e190b20748453178e8683c70cb0e4d0173ed07320e317974a40175524b5e5\": container with ID starting with 1a4e190b20748453178e8683c70cb0e4d0173ed07320e317974a40175524b5e5 not found: ID does not exist"
May 20 15:38:22.964966 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.964932    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f8d856c4d-wpxn8"]
May 20 15:38:22.972287 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:22.972260    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f8d856c4d-wpxn8"]
May 20 15:38:23.643716 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:38:23.643665    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae57088e-8522-4f76-91d3-06abe21d5577" path="/var/lib/kubelet/pods/ae57088e-8522-4f76-91d3-06abe21d5577/volumes"
May 20 15:39:38.972772 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:38.972738    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q"]
May 20 15:39:38.973220 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:38.973078    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae57088e-8522-4f76-91d3-06abe21d5577" containerName="console"
May 20 15:39:38.973220 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:38.973091    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae57088e-8522-4f76-91d3-06abe21d5577" containerName="console"
May 20 15:39:38.973220 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:38.973154    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="ae57088e-8522-4f76-91d3-06abe21d5577" containerName="console"
May 20 15:39:38.975347 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:38.975330    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q"
May 20 15:39:38.977724 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:38.977702    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 20 15:39:38.977848 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:38.977726    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 20 15:39:38.978504 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:38.978490    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-x4b4v\""
May 20 15:39:38.984409 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:38.984387    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q"]
May 20 15:39:39.051355 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:39.051320    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2fj9\" (UniqueName: \"kubernetes.io/projected/069f4a24-4843-4ee7-a8d7-c727277265ba-kube-api-access-n2fj9\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q\" (UID: \"069f4a24-4843-4ee7-a8d7-c727277265ba\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q"
May 20 15:39:39.051355 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:39.051356    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/069f4a24-4843-4ee7-a8d7-c727277265ba-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q\" (UID: \"069f4a24-4843-4ee7-a8d7-c727277265ba\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q"
May 20 15:39:39.051555 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:39.051470    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/069f4a24-4843-4ee7-a8d7-c727277265ba-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q\" (UID: \"069f4a24-4843-4ee7-a8d7-c727277265ba\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q"
May 20 15:39:39.152714 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:39.152661    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/069f4a24-4843-4ee7-a8d7-c727277265ba-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q\" (UID: \"069f4a24-4843-4ee7-a8d7-c727277265ba\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q"
May 20 15:39:39.152883 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:39.152740    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n2fj9\" (UniqueName: \"kubernetes.io/projected/069f4a24-4843-4ee7-a8d7-c727277265ba-kube-api-access-n2fj9\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q\" (UID: \"069f4a24-4843-4ee7-a8d7-c727277265ba\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q"
May 20 15:39:39.152883 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:39.152776    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/069f4a24-4843-4ee7-a8d7-c727277265ba-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q\" (UID: \"069f4a24-4843-4ee7-a8d7-c727277265ba\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q"
May 20 15:39:39.153197 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:39.153173    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/069f4a24-4843-4ee7-a8d7-c727277265ba-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q\" (UID: \"069f4a24-4843-4ee7-a8d7-c727277265ba\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q"
May 20 15:39:39.153244 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:39.153182    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/069f4a24-4843-4ee7-a8d7-c727277265ba-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q\" (UID: \"069f4a24-4843-4ee7-a8d7-c727277265ba\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q"
May 20 15:39:39.161958 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:39.161932    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2fj9\" (UniqueName: \"kubernetes.io/projected/069f4a24-4843-4ee7-a8d7-c727277265ba-kube-api-access-n2fj9\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q\" (UID: \"069f4a24-4843-4ee7-a8d7-c727277265ba\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q"
May 20 15:39:39.285213 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:39.285131    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q"
May 20 15:39:39.412982 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:39.412959    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q"]
May 20 15:39:39.415218 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:39:39.415185    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069f4a24_4843_4ee7_a8d7_c727277265ba.slice/crio-7b6e203c19f631c9e7bc9548e64513a7c22165895091b5e475c4812b43776093 WatchSource:0}: Error finding container 7b6e203c19f631c9e7bc9548e64513a7c22165895091b5e475c4812b43776093: Status 404 returned error can't find the container with id 7b6e203c19f631c9e7bc9548e64513a7c22165895091b5e475c4812b43776093
May 20 15:39:40.171787 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:40.171735    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q" event={"ID":"069f4a24-4843-4ee7-a8d7-c727277265ba","Type":"ContainerStarted","Data":"7b6e203c19f631c9e7bc9548e64513a7c22165895091b5e475c4812b43776093"}
May 20 15:39:46.189741 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:46.189644    2568 generic.go:358] "Generic (PLEG): container finished" podID="069f4a24-4843-4ee7-a8d7-c727277265ba" containerID="bb36123437ce65a6d464333a2f4aecfe8392d5dae3ae21663737bdab46817d1c" exitCode=0
May 20 15:39:46.189741 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:46.189724    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q" event={"ID":"069f4a24-4843-4ee7-a8d7-c727277265ba","Type":"ContainerDied","Data":"bb36123437ce65a6d464333a2f4aecfe8392d5dae3ae21663737bdab46817d1c"}
May 20 15:39:49.200463 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:49.200425    2568 generic.go:358] "Generic (PLEG): container finished" podID="069f4a24-4843-4ee7-a8d7-c727277265ba" containerID="c8f6b6af239ef6833585e74a411fbe6f2916c3c647c6f12edb21c97fdb5f7d74" exitCode=0
May 20 15:39:49.200945 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:49.200487    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q" event={"ID":"069f4a24-4843-4ee7-a8d7-c727277265ba","Type":"ContainerDied","Data":"c8f6b6af239ef6833585e74a411fbe6f2916c3c647c6f12edb21c97fdb5f7d74"}
May 20 15:39:55.225082 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:55.225034    2568 generic.go:358] "Generic (PLEG): container finished" podID="069f4a24-4843-4ee7-a8d7-c727277265ba" containerID="a2d4461014c044826a83551af3a14c710ec04d9abe073b5d261fddbfd1ac9708" exitCode=0
May 20 15:39:55.225520 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:55.225116    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q" event={"ID":"069f4a24-4843-4ee7-a8d7-c727277265ba","Type":"ContainerDied","Data":"a2d4461014c044826a83551af3a14c710ec04d9abe073b5d261fddbfd1ac9708"}
May 20 15:39:56.359270 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:56.359245    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q"
May 20 15:39:56.512988 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:56.512901    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/069f4a24-4843-4ee7-a8d7-c727277265ba-bundle\") pod \"069f4a24-4843-4ee7-a8d7-c727277265ba\" (UID: \"069f4a24-4843-4ee7-a8d7-c727277265ba\") "
May 20 15:39:56.513185 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:56.512992    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/069f4a24-4843-4ee7-a8d7-c727277265ba-util\") pod \"069f4a24-4843-4ee7-a8d7-c727277265ba\" (UID: \"069f4a24-4843-4ee7-a8d7-c727277265ba\") "
May 20 15:39:56.513185 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:56.513045    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2fj9\" (UniqueName: \"kubernetes.io/projected/069f4a24-4843-4ee7-a8d7-c727277265ba-kube-api-access-n2fj9\") pod \"069f4a24-4843-4ee7-a8d7-c727277265ba\" (UID: \"069f4a24-4843-4ee7-a8d7-c727277265ba\") "
May 20 15:39:56.513515 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:56.513490    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069f4a24-4843-4ee7-a8d7-c727277265ba-bundle" (OuterVolumeSpecName: "bundle") pod "069f4a24-4843-4ee7-a8d7-c727277265ba" (UID: "069f4a24-4843-4ee7-a8d7-c727277265ba"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:39:56.515481 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:56.515444    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069f4a24-4843-4ee7-a8d7-c727277265ba-kube-api-access-n2fj9" (OuterVolumeSpecName: "kube-api-access-n2fj9") pod "069f4a24-4843-4ee7-a8d7-c727277265ba" (UID: "069f4a24-4843-4ee7-a8d7-c727277265ba"). InnerVolumeSpecName "kube-api-access-n2fj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:39:56.518922 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:56.518902    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069f4a24-4843-4ee7-a8d7-c727277265ba-util" (OuterVolumeSpecName: "util") pod "069f4a24-4843-4ee7-a8d7-c727277265ba" (UID: "069f4a24-4843-4ee7-a8d7-c727277265ba"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:39:56.613919 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:56.613881    2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/069f4a24-4843-4ee7-a8d7-c727277265ba-bundle\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:39:56.613919 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:56.613912    2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/069f4a24-4843-4ee7-a8d7-c727277265ba-util\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:39:56.613919 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:56.613922    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2fj9\" (UniqueName: \"kubernetes.io/projected/069f4a24-4843-4ee7-a8d7-c727277265ba-kube-api-access-n2fj9\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:39:57.232245 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:57.232203    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q" event={"ID":"069f4a24-4843-4ee7-a8d7-c727277265ba","Type":"ContainerDied","Data":"7b6e203c19f631c9e7bc9548e64513a7c22165895091b5e475c4812b43776093"}
May 20 15:39:57.232245 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:57.232239    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b6e203c19f631c9e7bc9548e64513a7c22165895091b5e475c4812b43776093"
May 20 15:39:57.232457 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:39:57.232242    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cvbm7q"
May 20 15:40:01.530655 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:01.530618    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-ghjf6"]
May 20 15:40:01.531193 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:01.530990    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="069f4a24-4843-4ee7-a8d7-c727277265ba" containerName="util"
May 20 15:40:01.531193 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:01.531005    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="069f4a24-4843-4ee7-a8d7-c727277265ba" containerName="util"
May 20 15:40:01.531193 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:01.531032    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="069f4a24-4843-4ee7-a8d7-c727277265ba" containerName="pull"
May 20 15:40:01.531193 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:01.531041    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="069f4a24-4843-4ee7-a8d7-c727277265ba" containerName="pull"
May 20 15:40:01.531193 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:01.531054    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="069f4a24-4843-4ee7-a8d7-c727277265ba" containerName="extract"
May 20 15:40:01.531193 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:01.531062    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="069f4a24-4843-4ee7-a8d7-c727277265ba" containerName="extract"
May 20 15:40:01.531193 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:01.531134    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="069f4a24-4843-4ee7-a8d7-c727277265ba" containerName="extract"
May 20 15:40:01.533588 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:01.533562    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-ghjf6"
May 20 15:40:01.539965 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:01.539941    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
May 20 15:40:01.540148 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:01.540063    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
May 20 15:40:01.540148 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:01.540105    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-6wnb8\""
May 20 15:40:01.540289 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:01.540265    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
May 20 15:40:01.560867 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:01.560831    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-ghjf6"]
May 20 15:40:01.659444 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:01.659409    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29vgd\" (UniqueName: \"kubernetes.io/projected/2ec10a87-54a6-48b3-8d2a-0797627a852d-kube-api-access-29vgd\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-ghjf6\" (UID: \"2ec10a87-54a6-48b3-8d2a-0797627a852d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-ghjf6"
May 20 15:40:01.659611 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:01.659458    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/2ec10a87-54a6-48b3-8d2a-0797627a852d-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-ghjf6\" (UID: \"2ec10a87-54a6-48b3-8d2a-0797627a852d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-ghjf6"
May 20 15:40:01.759848 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:01.759811    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/2ec10a87-54a6-48b3-8d2a-0797627a852d-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-ghjf6\" (UID: \"2ec10a87-54a6-48b3-8d2a-0797627a852d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-ghjf6"
May 20 15:40:01.759991 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:01.759914    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-29vgd\" (UniqueName: \"kubernetes.io/projected/2ec10a87-54a6-48b3-8d2a-0797627a852d-kube-api-access-29vgd\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-ghjf6\" (UID: \"2ec10a87-54a6-48b3-8d2a-0797627a852d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-ghjf6"
May 20 15:40:01.762383 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:01.762359    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/2ec10a87-54a6-48b3-8d2a-0797627a852d-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-ghjf6\" (UID: \"2ec10a87-54a6-48b3-8d2a-0797627a852d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-ghjf6"
May 20 15:40:01.770241 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:01.770219    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29vgd\" (UniqueName: \"kubernetes.io/projected/2ec10a87-54a6-48b3-8d2a-0797627a852d-kube-api-access-29vgd\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-ghjf6\" (UID: \"2ec10a87-54a6-48b3-8d2a-0797627a852d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-ghjf6"
May 20 15:40:01.847170 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:01.847131    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-ghjf6"
May 20 15:40:01.989878 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:01.989854    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-ghjf6"]
May 20 15:40:01.992237 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:40:01.992206    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec10a87_54a6_48b3_8d2a_0797627a852d.slice/crio-eb01d073404b9499676d744b69a022c057dff8c3e60c7275e9f8cb5480e5b5e3 WatchSource:0}: Error finding container eb01d073404b9499676d744b69a022c057dff8c3e60c7275e9f8cb5480e5b5e3: Status 404 returned error can't find the container with id eb01d073404b9499676d744b69a022c057dff8c3e60c7275e9f8cb5480e5b5e3
May 20 15:40:02.248704 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:02.248613    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-ghjf6" event={"ID":"2ec10a87-54a6-48b3-8d2a-0797627a852d","Type":"ContainerStarted","Data":"eb01d073404b9499676d744b69a022c057dff8c3e60c7275e9f8cb5480e5b5e3"}
May 20 15:40:06.269046 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:06.268944    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-ghjf6" event={"ID":"2ec10a87-54a6-48b3-8d2a-0797627a852d","Type":"ContainerStarted","Data":"82ab7193b468ba0c64924892e292d494f0365c51e0b13c3372c4038b47bd7fa7"}
May 20 15:40:06.269429 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:06.269051    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-ghjf6"
May 20 15:40:06.292369 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:06.292321    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-ghjf6" podStartSLOduration=1.437535661 podStartE2EDuration="5.29230719s" podCreationTimestamp="2026-05-20 15:40:01 +0000 UTC" firstStartedPulling="2026-05-20 15:40:01.993761687 +0000 UTC m=+510.932603222" lastFinishedPulling="2026-05-20 15:40:05.848533217 +0000 UTC m=+514.787374751" observedRunningTime="2026-05-20 15:40:06.290219614 +0000 UTC m=+515.229061171" watchObservedRunningTime="2026-05-20 15:40:06.29230719 +0000 UTC m=+515.231148746"
May 20 15:40:06.473536 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:06.473501    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-w82tp"]
May 20 15:40:06.476622 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:06.476605    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-w82tp"
May 20 15:40:06.479200 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:06.479172    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
May 20 15:40:06.479200 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:06.479184    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-k6rd5\""
May 20 15:40:06.479495 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:06.479477    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
May 20 15:40:06.486864 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:06.486837    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-w82tp"]
May 20 15:40:06.598656 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:06.598609    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/a41d389e-9a3c-44ba-bc92-1d9884a28e15-cabundle0\") pod \"keda-operator-ffbb595cb-w82tp\" (UID: \"a41d389e-9a3c-44ba-bc92-1d9884a28e15\") " pod="openshift-keda/keda-operator-ffbb595cb-w82tp"
May 20 15:40:06.598821 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:06.598665    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a41d389e-9a3c-44ba-bc92-1d9884a28e15-certificates\") pod \"keda-operator-ffbb595cb-w82tp\" (UID: \"a41d389e-9a3c-44ba-bc92-1d9884a28e15\") " pod="openshift-keda/keda-operator-ffbb595cb-w82tp"
May 20 15:40:06.598821 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:06.598694    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wvj9\" (UniqueName: \"kubernetes.io/projected/a41d389e-9a3c-44ba-bc92-1d9884a28e15-kube-api-access-6wvj9\") pod \"keda-operator-ffbb595cb-w82tp\" (UID: \"a41d389e-9a3c-44ba-bc92-1d9884a28e15\") " pod="openshift-keda/keda-operator-ffbb595cb-w82tp"
May 20 15:40:06.699260 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:06.699225    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/a41d389e-9a3c-44ba-bc92-1d9884a28e15-cabundle0\") pod \"keda-operator-ffbb595cb-w82tp\" (UID: \"a41d389e-9a3c-44ba-bc92-1d9884a28e15\") " pod="openshift-keda/keda-operator-ffbb595cb-w82tp"
May 20 15:40:06.699446 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:06.699290    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a41d389e-9a3c-44ba-bc92-1d9884a28e15-certificates\") pod \"keda-operator-ffbb595cb-w82tp\" (UID: \"a41d389e-9a3c-44ba-bc92-1d9884a28e15\") " pod="openshift-keda/keda-operator-ffbb595cb-w82tp"
May 20 15:40:06.699446 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:06.699317    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6wvj9\" (UniqueName: \"kubernetes.io/projected/a41d389e-9a3c-44ba-bc92-1d9884a28e15-kube-api-access-6wvj9\") pod \"keda-operator-ffbb595cb-w82tp\" (UID: \"a41d389e-9a3c-44ba-bc92-1d9884a28e15\") " pod="openshift-keda/keda-operator-ffbb595cb-w82tp"
May 20 15:40:06.699564 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:40:06.699457    2568 secret.go:281] references non-existent secret key: ca.crt
May 20 15:40:06.699564 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:40:06.699473    2568 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 20 15:40:06.699564 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:40:06.699484    2568 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-w82tp: references non-existent secret key: ca.crt
May 20 15:40:06.699564 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:40:06.699545    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a41d389e-9a3c-44ba-bc92-1d9884a28e15-certificates podName:a41d389e-9a3c-44ba-bc92-1d9884a28e15 nodeName:}" failed. No retries permitted until 2026-05-20 15:40:07.199526172 +0000 UTC m=+516.138367709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a41d389e-9a3c-44ba-bc92-1d9884a28e15-certificates") pod "keda-operator-ffbb595cb-w82tp" (UID: "a41d389e-9a3c-44ba-bc92-1d9884a28e15") : references non-existent secret key: ca.crt
May 20 15:40:06.700071 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:06.700009    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/a41d389e-9a3c-44ba-bc92-1d9884a28e15-cabundle0\") pod \"keda-operator-ffbb595cb-w82tp\" (UID: \"a41d389e-9a3c-44ba-bc92-1d9884a28e15\") " pod="openshift-keda/keda-operator-ffbb595cb-w82tp"
May 20 15:40:06.712550 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:06.712523    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wvj9\" (UniqueName: \"kubernetes.io/projected/a41d389e-9a3c-44ba-bc92-1d9884a28e15-kube-api-access-6wvj9\") pod \"keda-operator-ffbb595cb-w82tp\" (UID: \"a41d389e-9a3c-44ba-bc92-1d9884a28e15\") " pod="openshift-keda/keda-operator-ffbb595cb-w82tp"
May 20 15:40:07.203706 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:07.203675    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a41d389e-9a3c-44ba-bc92-1d9884a28e15-certificates\") pod \"keda-operator-ffbb595cb-w82tp\" (UID: \"a41d389e-9a3c-44ba-bc92-1d9884a28e15\") " pod="openshift-keda/keda-operator-ffbb595cb-w82tp"
May 20 15:40:07.203882 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:40:07.203824    2568 secret.go:281] references non-existent secret key: ca.crt
May 20 15:40:07.203882 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:40:07.203843    2568 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 20 15:40:07.203882 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:40:07.203851    2568 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-w82tp: references non-existent secret key: ca.crt
May 20 15:40:07.203973 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:40:07.203905    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a41d389e-9a3c-44ba-bc92-1d9884a28e15-certificates podName:a41d389e-9a3c-44ba-bc92-1d9884a28e15 nodeName:}" failed. No retries permitted until 2026-05-20 15:40:08.203891032 +0000 UTC m=+517.142732568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a41d389e-9a3c-44ba-bc92-1d9884a28e15-certificates") pod "keda-operator-ffbb595cb-w82tp" (UID: "a41d389e-9a3c-44ba-bc92-1d9884a28e15") : references non-existent secret key: ca.crt
May 20 15:40:08.213168 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:08.213137    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a41d389e-9a3c-44ba-bc92-1d9884a28e15-certificates\") pod \"keda-operator-ffbb595cb-w82tp\" (UID: \"a41d389e-9a3c-44ba-bc92-1d9884a28e15\") " pod="openshift-keda/keda-operator-ffbb595cb-w82tp"
May 20 15:40:08.213551 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:40:08.213278    2568 secret.go:281] references non-existent secret key: ca.crt
May 20 15:40:08.213551 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:40:08.213301    2568 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 20 15:40:08.213551 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:40:08.213313    2568 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-w82tp: references non-existent secret key: ca.crt
May 20 15:40:08.213551 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:40:08.213365    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a41d389e-9a3c-44ba-bc92-1d9884a28e15-certificates podName:a41d389e-9a3c-44ba-bc92-1d9884a28e15 nodeName:}" failed. No retries permitted until 2026-05-20 15:40:10.213351743 +0000 UTC m=+519.152193283 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a41d389e-9a3c-44ba-bc92-1d9884a28e15-certificates") pod "keda-operator-ffbb595cb-w82tp" (UID: "a41d389e-9a3c-44ba-bc92-1d9884a28e15") : references non-existent secret key: ca.crt
May 20 15:40:10.229378 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:10.229345    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a41d389e-9a3c-44ba-bc92-1d9884a28e15-certificates\") pod \"keda-operator-ffbb595cb-w82tp\" (UID: \"a41d389e-9a3c-44ba-bc92-1d9884a28e15\") " pod="openshift-keda/keda-operator-ffbb595cb-w82tp"
May 20 15:40:10.231819 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:10.231800    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a41d389e-9a3c-44ba-bc92-1d9884a28e15-certificates\") pod \"keda-operator-ffbb595cb-w82tp\" (UID: \"a41d389e-9a3c-44ba-bc92-1d9884a28e15\") " pod="openshift-keda/keda-operator-ffbb595cb-w82tp"
May 20 15:40:10.388498 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:10.388456    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-w82tp"
May 20 15:40:10.526009 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:10.525985    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-w82tp"]
May 20 15:40:10.528658 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:40:10.528621    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41d389e_9a3c_44ba_bc92_1d9884a28e15.slice/crio-a6cdc0fc5e363d9774db1a56e708a2bc38a47710f9a8f1bfc33e8d79b57d5e74 WatchSource:0}: Error finding container a6cdc0fc5e363d9774db1a56e708a2bc38a47710f9a8f1bfc33e8d79b57d5e74: Status 404 returned error can't find the container with id a6cdc0fc5e363d9774db1a56e708a2bc38a47710f9a8f1bfc33e8d79b57d5e74
May 20 15:40:11.290292 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:11.290247    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-w82tp" event={"ID":"a41d389e-9a3c-44ba-bc92-1d9884a28e15","Type":"ContainerStarted","Data":"a6cdc0fc5e363d9774db1a56e708a2bc38a47710f9a8f1bfc33e8d79b57d5e74"}
May 20 15:40:14.301511 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:14.301479    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-w82tp" event={"ID":"a41d389e-9a3c-44ba-bc92-1d9884a28e15","Type":"ContainerStarted","Data":"03f2c8a81056ceb6d337d0436c4b3775e42415852e693da92fc1ce519c27e0af"}
May 20 15:40:14.301892 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:14.301615    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-ffbb595cb-w82tp"
May 20 15:40:14.322445 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:14.322399    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-operator-ffbb595cb-w82tp" podStartSLOduration=5.138918387 podStartE2EDuration="8.322384786s" podCreationTimestamp="2026-05-20 15:40:06 +0000 UTC" firstStartedPulling="2026-05-20 15:40:10.530132221 +0000 UTC m=+519.468973770" lastFinishedPulling="2026-05-20 15:40:13.71359863 +0000 UTC m=+522.652440169" observedRunningTime="2026-05-20 15:40:14.321740293 +0000 UTC m=+523.260581851" watchObservedRunningTime="2026-05-20 15:40:14.322384786 +0000 UTC m=+523.261226343"
May 20 15:40:27.275397 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:27.275367    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-ghjf6"
May 20 15:40:35.307419 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:40:35.307389    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-ffbb595cb-w82tp"
May 20 15:41:15.394550 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.394513    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-87b4b9896-qvcvk"]
May 20 15:41:15.397206 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.397187    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-87b4b9896-qvcvk"
May 20 15:41:15.399574 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.399541    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 20 15:41:15.399697 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.399578    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 20 15:41:15.399697 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.399576    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
May 20 15:41:15.400255 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.400234    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-vnjw9\""
May 20 15:41:15.411766 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.411743    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-87b4b9896-qvcvk"]
May 20 15:41:15.440712 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.440683    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-79f457656b-4pzks"]
May 20 15:41:15.443392 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.443375    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-4pzks"
May 20 15:41:15.445598 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.445578    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-qlbkl\""
May 20 15:41:15.445598 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.445577    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 20 15:41:15.451104 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.451064    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-4pzks"]
May 20 15:41:15.471638 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.471597    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bwkd\" (UniqueName: \"kubernetes.io/projected/6c778e7c-7d35-464d-ba7e-cf683631c232-kube-api-access-9bwkd\") pod \"kserve-controller-manager-87b4b9896-qvcvk\" (UID: \"6c778e7c-7d35-464d-ba7e-cf683631c232\") " pod="kserve/kserve-controller-manager-87b4b9896-qvcvk"
May 20 15:41:15.471807 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.471649    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c778e7c-7d35-464d-ba7e-cf683631c232-cert\") pod \"kserve-controller-manager-87b4b9896-qvcvk\" (UID: \"6c778e7c-7d35-464d-ba7e-cf683631c232\") " pod="kserve/kserve-controller-manager-87b4b9896-qvcvk"
May 20 15:41:15.572776 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.572737    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9bwkd\" (UniqueName: \"kubernetes.io/projected/6c778e7c-7d35-464d-ba7e-cf683631c232-kube-api-access-9bwkd\") pod \"kserve-controller-manager-87b4b9896-qvcvk\" (UID: \"6c778e7c-7d35-464d-ba7e-cf683631c232\") " pod="kserve/kserve-controller-manager-87b4b9896-qvcvk"
May 20 15:41:15.572953 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.572808    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c778e7c-7d35-464d-ba7e-cf683631c232-cert\") pod \"kserve-controller-manager-87b4b9896-qvcvk\" (UID: \"6c778e7c-7d35-464d-ba7e-cf683631c232\") " pod="kserve/kserve-controller-manager-87b4b9896-qvcvk"
May 20 15:41:15.572953 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.572842    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ddecec2b-9f2a-4742-963b-6d272fcb4308-data\") pod \"seaweedfs-79f457656b-4pzks\" (UID: \"ddecec2b-9f2a-4742-963b-6d272fcb4308\") " pod="kserve/seaweedfs-79f457656b-4pzks"
May 20 15:41:15.572953 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.572882    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drjgm\" (UniqueName: \"kubernetes.io/projected/ddecec2b-9f2a-4742-963b-6d272fcb4308-kube-api-access-drjgm\") pod \"seaweedfs-79f457656b-4pzks\" (UID: \"ddecec2b-9f2a-4742-963b-6d272fcb4308\") " pod="kserve/seaweedfs-79f457656b-4pzks"
May 20 15:41:15.575616 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.575590    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c778e7c-7d35-464d-ba7e-cf683631c232-cert\") pod \"kserve-controller-manager-87b4b9896-qvcvk\" (UID: \"6c778e7c-7d35-464d-ba7e-cf683631c232\") " pod="kserve/kserve-controller-manager-87b4b9896-qvcvk"
May 20 15:41:15.581665 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.581630    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bwkd\" (UniqueName: \"kubernetes.io/projected/6c778e7c-7d35-464d-ba7e-cf683631c232-kube-api-access-9bwkd\") pod \"kserve-controller-manager-87b4b9896-qvcvk\" (UID: \"6c778e7c-7d35-464d-ba7e-cf683631c232\") " pod="kserve/kserve-controller-manager-87b4b9896-qvcvk"
May 20 15:41:15.673825 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.673737    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ddecec2b-9f2a-4742-963b-6d272fcb4308-data\") pod \"seaweedfs-79f457656b-4pzks\" (UID: \"ddecec2b-9f2a-4742-963b-6d272fcb4308\") " pod="kserve/seaweedfs-79f457656b-4pzks"
May 20 15:41:15.673825 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.673783    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-drjgm\" (UniqueName: \"kubernetes.io/projected/ddecec2b-9f2a-4742-963b-6d272fcb4308-kube-api-access-drjgm\") pod \"seaweedfs-79f457656b-4pzks\" (UID: \"ddecec2b-9f2a-4742-963b-6d272fcb4308\") " pod="kserve/seaweedfs-79f457656b-4pzks"
May 20 15:41:15.674156 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.674134    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ddecec2b-9f2a-4742-963b-6d272fcb4308-data\") pod \"seaweedfs-79f457656b-4pzks\" (UID: \"ddecec2b-9f2a-4742-963b-6d272fcb4308\") " pod="kserve/seaweedfs-79f457656b-4pzks"
May 20 15:41:15.688399 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.688375    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drjgm\" (UniqueName: \"kubernetes.io/projected/ddecec2b-9f2a-4742-963b-6d272fcb4308-kube-api-access-drjgm\") pod \"seaweedfs-79f457656b-4pzks\" (UID: \"ddecec2b-9f2a-4742-963b-6d272fcb4308\") " pod="kserve/seaweedfs-79f457656b-4pzks"
May 20 15:41:15.708210 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.708184    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-87b4b9896-qvcvk"
May 20 15:41:15.754170 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.754137    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-4pzks"
May 20 15:41:15.850629 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.850600    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-87b4b9896-qvcvk"]
May 20 15:41:15.853840 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:41:15.853800    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c778e7c_7d35_464d_ba7e_cf683631c232.slice/crio-2db22ee5306e81d27ef979e255170bec0564468bc39743239c8b83b5f7f4cce5 WatchSource:0}: Error finding container 2db22ee5306e81d27ef979e255170bec0564468bc39743239c8b83b5f7f4cce5: Status 404 returned error can't find the container with id 2db22ee5306e81d27ef979e255170bec0564468bc39743239c8b83b5f7f4cce5
May 20 15:41:15.896334 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:15.896179    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-4pzks"]
May 20 15:41:15.898485 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:41:15.898458    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddecec2b_9f2a_4742_963b_6d272fcb4308.slice/crio-37c355d1af6b24eea1547b3d348d78f229d1afd95adca00e7ace13ba822241fa WatchSource:0}: Error finding container 37c355d1af6b24eea1547b3d348d78f229d1afd95adca00e7ace13ba822241fa: Status 404 returned error can't find the container with id 37c355d1af6b24eea1547b3d348d78f229d1afd95adca00e7ace13ba822241fa
May 20 15:41:16.523216 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:16.523132    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-87b4b9896-qvcvk" event={"ID":"6c778e7c-7d35-464d-ba7e-cf683631c232","Type":"ContainerStarted","Data":"2db22ee5306e81d27ef979e255170bec0564468bc39743239c8b83b5f7f4cce5"}
May 20 15:41:16.525075 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:16.524978    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-4pzks" event={"ID":"ddecec2b-9f2a-4742-963b-6d272fcb4308","Type":"ContainerStarted","Data":"37c355d1af6b24eea1547b3d348d78f229d1afd95adca00e7ace13ba822241fa"}
May 20 15:41:20.544698 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:20.544662    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-87b4b9896-qvcvk" event={"ID":"6c778e7c-7d35-464d-ba7e-cf683631c232","Type":"ContainerStarted","Data":"0e018641399cfccf416b5645f8a2f8300a50a34bad55a8c2394c4a70449a5d96"}
May 20 15:41:20.545174 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:20.544727    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-87b4b9896-qvcvk"
May 20 15:41:20.546011 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:20.545988    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-4pzks" event={"ID":"ddecec2b-9f2a-4742-963b-6d272fcb4308","Type":"ContainerStarted","Data":"3030db05bd25e21e971dd0d094e3cab4729f7b039eb05ff18685f41e31f4d7bd"}
May 20 15:41:20.546126 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:20.546060    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-79f457656b-4pzks"
May 20 15:41:20.563413 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:20.563358    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-87b4b9896-qvcvk" podStartSLOduration=1.976059 podStartE2EDuration="5.563343948s" podCreationTimestamp="2026-05-20 15:41:15 +0000 UTC" firstStartedPulling="2026-05-20 15:41:15.855313506 +0000 UTC m=+584.794155041" lastFinishedPulling="2026-05-20 15:41:19.442598452 +0000 UTC m=+588.381439989" observedRunningTime="2026-05-20 15:41:20.562724764 +0000 UTC m=+589.501566320" watchObservedRunningTime="2026-05-20 15:41:20.563343948 +0000 UTC m=+589.502185505"
May 20 15:41:20.580652 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:20.580604    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-79f457656b-4pzks" podStartSLOduration=1.968621781 podStartE2EDuration="5.580589457s" podCreationTimestamp="2026-05-20 15:41:15 +0000 UTC" firstStartedPulling="2026-05-20 15:41:15.899745058 +0000 UTC m=+584.838586597" lastFinishedPulling="2026-05-20 15:41:19.511712733 +0000 UTC m=+588.450554273" observedRunningTime="2026-05-20 15:41:20.579716887 +0000 UTC m=+589.518558449" watchObservedRunningTime="2026-05-20 15:41:20.580589457 +0000 UTC m=+589.519431014"
May 20 15:41:21.402053 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.402002    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-b95c64d84-z69xn"]
May 20 15:41:21.404917 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.404900    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.417937 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.417911    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b95c64d84-z69xn"]
May 20 15:41:21.529650 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.529615    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ecacce7e-932c-4c2a-9108-78652b56bcd8-console-oauth-config\") pod \"console-b95c64d84-z69xn\" (UID: \"ecacce7e-932c-4c2a-9108-78652b56bcd8\") " pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.529650 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.529653    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ecacce7e-932c-4c2a-9108-78652b56bcd8-service-ca\") pod \"console-b95c64d84-z69xn\" (UID: \"ecacce7e-932c-4c2a-9108-78652b56bcd8\") " pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.529876 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.529678    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecacce7e-932c-4c2a-9108-78652b56bcd8-trusted-ca-bundle\") pod \"console-b95c64d84-z69xn\" (UID: \"ecacce7e-932c-4c2a-9108-78652b56bcd8\") " pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.529876 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.529736    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgqs8\" (UniqueName: \"kubernetes.io/projected/ecacce7e-932c-4c2a-9108-78652b56bcd8-kube-api-access-jgqs8\") pod \"console-b95c64d84-z69xn\" (UID: \"ecacce7e-932c-4c2a-9108-78652b56bcd8\") " pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.529876 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.529814    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ecacce7e-932c-4c2a-9108-78652b56bcd8-console-serving-cert\") pod \"console-b95c64d84-z69xn\" (UID: \"ecacce7e-932c-4c2a-9108-78652b56bcd8\") " pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.529876 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.529857    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ecacce7e-932c-4c2a-9108-78652b56bcd8-oauth-serving-cert\") pod \"console-b95c64d84-z69xn\" (UID: \"ecacce7e-932c-4c2a-9108-78652b56bcd8\") " pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.529998 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.529885    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ecacce7e-932c-4c2a-9108-78652b56bcd8-console-config\") pod \"console-b95c64d84-z69xn\" (UID: \"ecacce7e-932c-4c2a-9108-78652b56bcd8\") " pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.630736 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.630698    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ecacce7e-932c-4c2a-9108-78652b56bcd8-console-oauth-config\") pod \"console-b95c64d84-z69xn\" (UID: \"ecacce7e-932c-4c2a-9108-78652b56bcd8\") " pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.630736 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.630735    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ecacce7e-932c-4c2a-9108-78652b56bcd8-service-ca\") pod \"console-b95c64d84-z69xn\" (UID: \"ecacce7e-932c-4c2a-9108-78652b56bcd8\") " pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.631312 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.630753    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecacce7e-932c-4c2a-9108-78652b56bcd8-trusted-ca-bundle\") pod \"console-b95c64d84-z69xn\" (UID: \"ecacce7e-932c-4c2a-9108-78652b56bcd8\") " pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.631312 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.630769    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jgqs8\" (UniqueName: \"kubernetes.io/projected/ecacce7e-932c-4c2a-9108-78652b56bcd8-kube-api-access-jgqs8\") pod \"console-b95c64d84-z69xn\" (UID: \"ecacce7e-932c-4c2a-9108-78652b56bcd8\") " pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.631312 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.630859    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ecacce7e-932c-4c2a-9108-78652b56bcd8-console-serving-cert\") pod \"console-b95c64d84-z69xn\" (UID: \"ecacce7e-932c-4c2a-9108-78652b56bcd8\") " pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.631312 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.630923    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ecacce7e-932c-4c2a-9108-78652b56bcd8-oauth-serving-cert\") pod \"console-b95c64d84-z69xn\" (UID: \"ecacce7e-932c-4c2a-9108-78652b56bcd8\") " pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.631312 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.630966    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ecacce7e-932c-4c2a-9108-78652b56bcd8-console-config\") pod \"console-b95c64d84-z69xn\" (UID: \"ecacce7e-932c-4c2a-9108-78652b56bcd8\") " pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.631607 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.631564    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ecacce7e-932c-4c2a-9108-78652b56bcd8-service-ca\") pod \"console-b95c64d84-z69xn\" (UID: \"ecacce7e-932c-4c2a-9108-78652b56bcd8\") " pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.631769 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.631747    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecacce7e-932c-4c2a-9108-78652b56bcd8-trusted-ca-bundle\") pod \"console-b95c64d84-z69xn\" (UID: \"ecacce7e-932c-4c2a-9108-78652b56bcd8\") " pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.631808 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.631778    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ecacce7e-932c-4c2a-9108-78652b56bcd8-console-config\") pod \"console-b95c64d84-z69xn\" (UID: \"ecacce7e-932c-4c2a-9108-78652b56bcd8\") " pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.631880 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.631859    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ecacce7e-932c-4c2a-9108-78652b56bcd8-oauth-serving-cert\") pod \"console-b95c64d84-z69xn\" (UID: \"ecacce7e-932c-4c2a-9108-78652b56bcd8\") " pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.633620 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.633600    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ecacce7e-932c-4c2a-9108-78652b56bcd8-console-oauth-config\") pod \"console-b95c64d84-z69xn\" (UID: \"ecacce7e-932c-4c2a-9108-78652b56bcd8\") " pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.633686 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.633621    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ecacce7e-932c-4c2a-9108-78652b56bcd8-console-serving-cert\") pod \"console-b95c64d84-z69xn\" (UID: \"ecacce7e-932c-4c2a-9108-78652b56bcd8\") " pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.640479 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.640451    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgqs8\" (UniqueName: \"kubernetes.io/projected/ecacce7e-932c-4c2a-9108-78652b56bcd8-kube-api-access-jgqs8\") pod \"console-b95c64d84-z69xn\" (UID: \"ecacce7e-932c-4c2a-9108-78652b56bcd8\") " pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.713642 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.713555    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:21.844204 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:21.844068    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b95c64d84-z69xn"]
May 20 15:41:21.846535 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:41:21.846506    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecacce7e_932c_4c2a_9108_78652b56bcd8.slice/crio-51a65c4ac4b2dd5c4b2a5a2b60e109b6277bf036fccf391424189cf91995bd21 WatchSource:0}: Error finding container 51a65c4ac4b2dd5c4b2a5a2b60e109b6277bf036fccf391424189cf91995bd21: Status 404 returned error can't find the container with id 51a65c4ac4b2dd5c4b2a5a2b60e109b6277bf036fccf391424189cf91995bd21
May 20 15:41:22.553500 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:22.553446    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b95c64d84-z69xn" event={"ID":"ecacce7e-932c-4c2a-9108-78652b56bcd8","Type":"ContainerStarted","Data":"cf69f52792fc256ebd2453c78648c1fc6cdbabf37e5bf3813ab3a406e6eb8571"}
May 20 15:41:22.553500 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:22.553501    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b95c64d84-z69xn" event={"ID":"ecacce7e-932c-4c2a-9108-78652b56bcd8","Type":"ContainerStarted","Data":"51a65c4ac4b2dd5c4b2a5a2b60e109b6277bf036fccf391424189cf91995bd21"}
May 20 15:41:22.573338 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:22.573272    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-b95c64d84-z69xn" podStartSLOduration=1.573252891 podStartE2EDuration="1.573252891s" podCreationTimestamp="2026-05-20 15:41:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:41:22.57214775 +0000 UTC m=+591.510989303" watchObservedRunningTime="2026-05-20 15:41:22.573252891 +0000 UTC m=+591.512094448"
May 20 15:41:26.552070 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:26.552012    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-79f457656b-4pzks"
May 20 15:41:31.590536 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:31.590506    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 15:41:31.591037 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:31.591002    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 15:41:31.713711 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:31.713680    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:31.713711 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:31.713730    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:31.719273 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:31.719249    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:32.590228 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:32.590198    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b95c64d84-z69xn"
May 20 15:41:32.640547 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:32.640518    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58d54fb74b-ldhr7"]
May 20 15:41:50.788319 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:50.788285    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-87b4b9896-qvcvk"]
May 20 15:41:50.789999 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:50.788513    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/kserve-controller-manager-87b4b9896-qvcvk" podUID="6c778e7c-7d35-464d-ba7e-cf683631c232" containerName="manager" containerID="cri-o://0e018641399cfccf416b5645f8a2f8300a50a34bad55a8c2394c4a70449a5d96" gracePeriod=10
May 20 15:41:50.793595 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:50.793572    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-87b4b9896-qvcvk"
May 20 15:41:50.813911 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:50.813877    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-87b4b9896-c4cbk"]
May 20 15:41:50.816928 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:50.816909    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-87b4b9896-c4cbk"
May 20 15:41:50.827622 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:50.827593    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-87b4b9896-c4cbk"]
May 20 15:41:50.887277 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:50.887240    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cj6g\" (UniqueName: \"kubernetes.io/projected/c9af9950-cb53-4e78-a702-7cc8b21f4472-kube-api-access-6cj6g\") pod \"kserve-controller-manager-87b4b9896-c4cbk\" (UID: \"c9af9950-cb53-4e78-a702-7cc8b21f4472\") " pod="kserve/kserve-controller-manager-87b4b9896-c4cbk"
May 20 15:41:50.887433 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:50.887298    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9af9950-cb53-4e78-a702-7cc8b21f4472-cert\") pod \"kserve-controller-manager-87b4b9896-c4cbk\" (UID: \"c9af9950-cb53-4e78-a702-7cc8b21f4472\") " pod="kserve/kserve-controller-manager-87b4b9896-c4cbk"
May 20 15:41:50.987888 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:50.987855    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6cj6g\" (UniqueName: \"kubernetes.io/projected/c9af9950-cb53-4e78-a702-7cc8b21f4472-kube-api-access-6cj6g\") pod \"kserve-controller-manager-87b4b9896-c4cbk\" (UID: \"c9af9950-cb53-4e78-a702-7cc8b21f4472\") " pod="kserve/kserve-controller-manager-87b4b9896-c4cbk"
May 20 15:41:50.988098 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:50.987902    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9af9950-cb53-4e78-a702-7cc8b21f4472-cert\") pod \"kserve-controller-manager-87b4b9896-c4cbk\" (UID: \"c9af9950-cb53-4e78-a702-7cc8b21f4472\") " pod="kserve/kserve-controller-manager-87b4b9896-c4cbk"
May 20 15:41:50.990738 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:50.990708    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9af9950-cb53-4e78-a702-7cc8b21f4472-cert\") pod \"kserve-controller-manager-87b4b9896-c4cbk\" (UID: \"c9af9950-cb53-4e78-a702-7cc8b21f4472\") " pod="kserve/kserve-controller-manager-87b4b9896-c4cbk"
May 20 15:41:50.997985 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:50.997961    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cj6g\" (UniqueName: \"kubernetes.io/projected/c9af9950-cb53-4e78-a702-7cc8b21f4472-kube-api-access-6cj6g\") pod \"kserve-controller-manager-87b4b9896-c4cbk\" (UID: \"c9af9950-cb53-4e78-a702-7cc8b21f4472\") " pod="kserve/kserve-controller-manager-87b4b9896-c4cbk"
May 20 15:41:51.039248 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:51.039185    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-87b4b9896-qvcvk"
May 20 15:41:51.088765 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:51.088733    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c778e7c-7d35-464d-ba7e-cf683631c232-cert\") pod \"6c778e7c-7d35-464d-ba7e-cf683631c232\" (UID: \"6c778e7c-7d35-464d-ba7e-cf683631c232\") "
May 20 15:41:51.088960 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:51.088790    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bwkd\" (UniqueName: \"kubernetes.io/projected/6c778e7c-7d35-464d-ba7e-cf683631c232-kube-api-access-9bwkd\") pod \"6c778e7c-7d35-464d-ba7e-cf683631c232\" (UID: \"6c778e7c-7d35-464d-ba7e-cf683631c232\") "
May 20 15:41:51.091196 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:51.091169    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c778e7c-7d35-464d-ba7e-cf683631c232-kube-api-access-9bwkd" (OuterVolumeSpecName: "kube-api-access-9bwkd") pod "6c778e7c-7d35-464d-ba7e-cf683631c232" (UID: "6c778e7c-7d35-464d-ba7e-cf683631c232"). InnerVolumeSpecName "kube-api-access-9bwkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:41:51.091316 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:51.091249    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c778e7c-7d35-464d-ba7e-cf683631c232-cert" (OuterVolumeSpecName: "cert") pod "6c778e7c-7d35-464d-ba7e-cf683631c232" (UID: "6c778e7c-7d35-464d-ba7e-cf683631c232"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:41:51.161392 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:51.161353    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-87b4b9896-c4cbk"
May 20 15:41:51.189391 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:51.189359    2568 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c778e7c-7d35-464d-ba7e-cf683631c232-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:41:51.189391 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:51.189390    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bwkd\" (UniqueName: \"kubernetes.io/projected/6c778e7c-7d35-464d-ba7e-cf683631c232-kube-api-access-9bwkd\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:41:51.307681 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:51.307656    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-87b4b9896-c4cbk"]
May 20 15:41:51.309233 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:41:51.309208    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9af9950_cb53_4e78_a702_7cc8b21f4472.slice/crio-35f5a12359eb2563003931f7a42eacbc51d2a2c653e5511b0fba35a3ca7e1a62 WatchSource:0}: Error finding container 35f5a12359eb2563003931f7a42eacbc51d2a2c653e5511b0fba35a3ca7e1a62: Status 404 returned error can't find the container with id 35f5a12359eb2563003931f7a42eacbc51d2a2c653e5511b0fba35a3ca7e1a62
May 20 15:41:51.651265 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:51.651234    2568 generic.go:358] "Generic (PLEG): container finished" podID="6c778e7c-7d35-464d-ba7e-cf683631c232" containerID="0e018641399cfccf416b5645f8a2f8300a50a34bad55a8c2394c4a70449a5d96" exitCode=0
May 20 15:41:51.651403 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:51.651328    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-87b4b9896-qvcvk"
May 20 15:41:51.651475 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:51.651320    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-87b4b9896-qvcvk" event={"ID":"6c778e7c-7d35-464d-ba7e-cf683631c232","Type":"ContainerDied","Data":"0e018641399cfccf416b5645f8a2f8300a50a34bad55a8c2394c4a70449a5d96"}
May 20 15:41:51.651584 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:51.651483    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-87b4b9896-qvcvk" event={"ID":"6c778e7c-7d35-464d-ba7e-cf683631c232","Type":"ContainerDied","Data":"2db22ee5306e81d27ef979e255170bec0564468bc39743239c8b83b5f7f4cce5"}
May 20 15:41:51.651584 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:51.651509    2568 scope.go:117] "RemoveContainer" containerID="0e018641399cfccf416b5645f8a2f8300a50a34bad55a8c2394c4a70449a5d96"
May 20 15:41:51.652628 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:51.652597    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-87b4b9896-c4cbk" event={"ID":"c9af9950-cb53-4e78-a702-7cc8b21f4472","Type":"ContainerStarted","Data":"35f5a12359eb2563003931f7a42eacbc51d2a2c653e5511b0fba35a3ca7e1a62"}
May 20 15:41:51.670185 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:51.670141    2568 scope.go:117] "RemoveContainer" containerID="0e018641399cfccf416b5645f8a2f8300a50a34bad55a8c2394c4a70449a5d96"
May 20 15:41:51.670499 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:41:51.670474    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e018641399cfccf416b5645f8a2f8300a50a34bad55a8c2394c4a70449a5d96\": container with ID starting with 0e018641399cfccf416b5645f8a2f8300a50a34bad55a8c2394c4a70449a5d96 not found: ID does not exist" containerID="0e018641399cfccf416b5645f8a2f8300a50a34bad55a8c2394c4a70449a5d96"
May 20 15:41:51.670597 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:51.670512    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e018641399cfccf416b5645f8a2f8300a50a34bad55a8c2394c4a70449a5d96"} err="failed to get container status \"0e018641399cfccf416b5645f8a2f8300a50a34bad55a8c2394c4a70449a5d96\": rpc error: code = NotFound desc = could not find container \"0e018641399cfccf416b5645f8a2f8300a50a34bad55a8c2394c4a70449a5d96\": container with ID starting with 0e018641399cfccf416b5645f8a2f8300a50a34bad55a8c2394c4a70449a5d96 not found: ID does not exist"
May 20 15:41:51.692835 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:51.692795    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-87b4b9896-qvcvk"]
May 20 15:41:51.697593 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:51.697566    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-87b4b9896-qvcvk"]
May 20 15:41:52.657233 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:52.657196    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-87b4b9896-c4cbk" event={"ID":"c9af9950-cb53-4e78-a702-7cc8b21f4472","Type":"ContainerStarted","Data":"75722146b771197e78add40a769a9c81c77a6b7f40482bf2c0e3d3401b7080f8"}
May 20 15:41:52.657700 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:52.657263    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-87b4b9896-c4cbk"
May 20 15:41:52.677088 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:52.676999    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-87b4b9896-c4cbk" podStartSLOduration=2.365245895 podStartE2EDuration="2.676980415s" podCreationTimestamp="2026-05-20 15:41:50 +0000 UTC" firstStartedPulling="2026-05-20 15:41:51.310487587 +0000 UTC m=+620.249329122" lastFinishedPulling="2026-05-20 15:41:51.622222105 +0000 UTC m=+620.561063642" observedRunningTime="2026-05-20 15:41:52.675210655 +0000 UTC m=+621.614052211" watchObservedRunningTime="2026-05-20 15:41:52.676980415 +0000 UTC m=+621.615821973"
May 20 15:41:53.644452 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:53.644408    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c778e7c-7d35-464d-ba7e-cf683631c232" path="/var/lib/kubelet/pods/6c778e7c-7d35-464d-ba7e-cf683631c232/volumes"
May 20 15:41:57.660647 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:57.660610    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-58d54fb74b-ldhr7" podUID="a4a13f6a-33c1-481d-a27a-82931348c609" containerName="console" containerID="cri-o://a5d58e65cd045a96a967d3598c7e90b3c7e2a6bc76ada96574dd734ae9bce832" gracePeriod=15
May 20 15:41:57.909225 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:57.909202    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-58d54fb74b-ldhr7_a4a13f6a-33c1-481d-a27a-82931348c609/console/0.log"
May 20 15:41:57.909350 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:57.909265    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:41:57.944147 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:57.944061    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j58g2\" (UniqueName: \"kubernetes.io/projected/a4a13f6a-33c1-481d-a27a-82931348c609-kube-api-access-j58g2\") pod \"a4a13f6a-33c1-481d-a27a-82931348c609\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") "
May 20 15:41:57.944147 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:57.944125    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-console-config\") pod \"a4a13f6a-33c1-481d-a27a-82931348c609\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") "
May 20 15:41:57.944363 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:57.944229    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4a13f6a-33c1-481d-a27a-82931348c609-console-serving-cert\") pod \"a4a13f6a-33c1-481d-a27a-82931348c609\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") "
May 20 15:41:57.944363 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:57.944267    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-trusted-ca-bundle\") pod \"a4a13f6a-33c1-481d-a27a-82931348c609\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") "
May 20 15:41:57.944363 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:57.944294    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-oauth-serving-cert\") pod \"a4a13f6a-33c1-481d-a27a-82931348c609\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") "
May 20 15:41:57.944363 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:57.944344    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-service-ca\") pod \"a4a13f6a-33c1-481d-a27a-82931348c609\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") "
May 20 15:41:57.944561 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:57.944372    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4a13f6a-33c1-481d-a27a-82931348c609-console-oauth-config\") pod \"a4a13f6a-33c1-481d-a27a-82931348c609\" (UID: \"a4a13f6a-33c1-481d-a27a-82931348c609\") "
May 20 15:41:57.944661 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:57.944636    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-console-config" (OuterVolumeSpecName: "console-config") pod "a4a13f6a-33c1-481d-a27a-82931348c609" (UID: "a4a13f6a-33c1-481d-a27a-82931348c609"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:41:57.944730 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:57.944644    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a4a13f6a-33c1-481d-a27a-82931348c609" (UID: "a4a13f6a-33c1-481d-a27a-82931348c609"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:41:57.944781 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:57.944746    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a4a13f6a-33c1-481d-a27a-82931348c609" (UID: "a4a13f6a-33c1-481d-a27a-82931348c609"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:41:57.944781 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:57.944762    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-service-ca" (OuterVolumeSpecName: "service-ca") pod "a4a13f6a-33c1-481d-a27a-82931348c609" (UID: "a4a13f6a-33c1-481d-a27a-82931348c609"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:41:57.946618 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:57.946589    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4a13f6a-33c1-481d-a27a-82931348c609-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a4a13f6a-33c1-481d-a27a-82931348c609" (UID: "a4a13f6a-33c1-481d-a27a-82931348c609"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:41:57.946769 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:57.946749    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4a13f6a-33c1-481d-a27a-82931348c609-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a4a13f6a-33c1-481d-a27a-82931348c609" (UID: "a4a13f6a-33c1-481d-a27a-82931348c609"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:41:57.946809 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:57.946760    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a13f6a-33c1-481d-a27a-82931348c609-kube-api-access-j58g2" (OuterVolumeSpecName: "kube-api-access-j58g2") pod "a4a13f6a-33c1-481d-a27a-82931348c609" (UID: "a4a13f6a-33c1-481d-a27a-82931348c609"). InnerVolumeSpecName "kube-api-access-j58g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:41:58.045538 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:58.045503    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4a13f6a-33c1-481d-a27a-82931348c609-console-serving-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:41:58.045538 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:58.045532    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-trusted-ca-bundle\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:41:58.045538 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:58.045542    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-oauth-serving-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:41:58.045769 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:58.045556    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-service-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:41:58.045769 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:58.045566    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4a13f6a-33c1-481d-a27a-82931348c609-console-oauth-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:41:58.045769 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:58.045575    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j58g2\" (UniqueName: \"kubernetes.io/projected/a4a13f6a-33c1-481d-a27a-82931348c609-kube-api-access-j58g2\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:41:58.045769 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:58.045584    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4a13f6a-33c1-481d-a27a-82931348c609-console-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:41:58.678293 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:58.678264    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-58d54fb74b-ldhr7_a4a13f6a-33c1-481d-a27a-82931348c609/console/0.log"
May 20 15:41:58.678676 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:58.678307    2568 generic.go:358] "Generic (PLEG): container finished" podID="a4a13f6a-33c1-481d-a27a-82931348c609" containerID="a5d58e65cd045a96a967d3598c7e90b3c7e2a6bc76ada96574dd734ae9bce832" exitCode=2
May 20 15:41:58.678676 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:58.678342    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d54fb74b-ldhr7" event={"ID":"a4a13f6a-33c1-481d-a27a-82931348c609","Type":"ContainerDied","Data":"a5d58e65cd045a96a967d3598c7e90b3c7e2a6bc76ada96574dd734ae9bce832"}
May 20 15:41:58.678676 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:58.678368    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d54fb74b-ldhr7" event={"ID":"a4a13f6a-33c1-481d-a27a-82931348c609","Type":"ContainerDied","Data":"54fede5c838ad68d38bdd9924e2d1bdb4baac034a402daef37f0df50f9566743"}
May 20 15:41:58.678676 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:58.678382    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d54fb74b-ldhr7"
May 20 15:41:58.678676 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:58.678390    2568 scope.go:117] "RemoveContainer" containerID="a5d58e65cd045a96a967d3598c7e90b3c7e2a6bc76ada96574dd734ae9bce832"
May 20 15:41:58.687157 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:58.687134    2568 scope.go:117] "RemoveContainer" containerID="a5d58e65cd045a96a967d3598c7e90b3c7e2a6bc76ada96574dd734ae9bce832"
May 20 15:41:58.687426 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:41:58.687405    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5d58e65cd045a96a967d3598c7e90b3c7e2a6bc76ada96574dd734ae9bce832\": container with ID starting with a5d58e65cd045a96a967d3598c7e90b3c7e2a6bc76ada96574dd734ae9bce832 not found: ID does not exist" containerID="a5d58e65cd045a96a967d3598c7e90b3c7e2a6bc76ada96574dd734ae9bce832"
May 20 15:41:58.687475 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:58.687436    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5d58e65cd045a96a967d3598c7e90b3c7e2a6bc76ada96574dd734ae9bce832"} err="failed to get container status \"a5d58e65cd045a96a967d3598c7e90b3c7e2a6bc76ada96574dd734ae9bce832\": rpc error: code = NotFound desc = could not find container \"a5d58e65cd045a96a967d3598c7e90b3c7e2a6bc76ada96574dd734ae9bce832\": container with ID starting with a5d58e65cd045a96a967d3598c7e90b3c7e2a6bc76ada96574dd734ae9bce832 not found: ID does not exist"
May 20 15:41:58.698860 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:58.698831    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58d54fb74b-ldhr7"]
May 20 15:41:58.702668 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:58.702646    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58d54fb74b-ldhr7"]
May 20 15:41:59.644036 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:41:59.643985    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4a13f6a-33c1-481d-a27a-82931348c609" path="/var/lib/kubelet/pods/a4a13f6a-33c1-481d-a27a-82931348c609/volumes"
May 20 15:42:23.665978 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:23.665948    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-87b4b9896-c4cbk"
May 20 15:42:41.483119 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:41.483087    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-pzwkl"]
May 20 15:42:41.483505 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:41.483420    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c778e7c-7d35-464d-ba7e-cf683631c232" containerName="manager"
May 20 15:42:41.483505 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:41.483431    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c778e7c-7d35-464d-ba7e-cf683631c232" containerName="manager"
May 20 15:42:41.483505 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:41.483440    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4a13f6a-33c1-481d-a27a-82931348c609" containerName="console"
May 20 15:42:41.483505 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:41.483445    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4a13f6a-33c1-481d-a27a-82931348c609" containerName="console"
May 20 15:42:41.483505 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:41.483501    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="a4a13f6a-33c1-481d-a27a-82931348c609" containerName="console"
May 20 15:42:41.483666 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:41.483512    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="6c778e7c-7d35-464d-ba7e-cf683631c232" containerName="manager"
May 20 15:42:41.486771 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:41.486751    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pzwkl"
May 20 15:42:41.494714 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:41.494691    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-pzwkl"]
May 20 15:42:41.524874 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:41.524839    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlrt2\" (UniqueName: \"kubernetes.io/projected/e006510d-31f4-435c-9001-eaae8ed6b5e5-kube-api-access-wlrt2\") pod \"s3-init-pzwkl\" (UID: \"e006510d-31f4-435c-9001-eaae8ed6b5e5\") " pod="kserve/s3-init-pzwkl"
May 20 15:42:41.626148 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:41.626108    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wlrt2\" (UniqueName: \"kubernetes.io/projected/e006510d-31f4-435c-9001-eaae8ed6b5e5-kube-api-access-wlrt2\") pod \"s3-init-pzwkl\" (UID: \"e006510d-31f4-435c-9001-eaae8ed6b5e5\") " pod="kserve/s3-init-pzwkl"
May 20 15:42:41.637900 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:41.637874    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlrt2\" (UniqueName: \"kubernetes.io/projected/e006510d-31f4-435c-9001-eaae8ed6b5e5-kube-api-access-wlrt2\") pod \"s3-init-pzwkl\" (UID: \"e006510d-31f4-435c-9001-eaae8ed6b5e5\") " pod="kserve/s3-init-pzwkl"
May 20 15:42:41.804606 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:41.804579    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pzwkl"
May 20 15:42:41.933890 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:41.933868    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-pzwkl"]
May 20 15:42:41.936456 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:42:41.936426    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode006510d_31f4_435c_9001_eaae8ed6b5e5.slice/crio-56e6b5b3584daefb0fa84f8bbd2e90c6027d0c1749b5b19a8eb28987ad6cc734 WatchSource:0}: Error finding container 56e6b5b3584daefb0fa84f8bbd2e90c6027d0c1749b5b19a8eb28987ad6cc734: Status 404 returned error can't find the container with id 56e6b5b3584daefb0fa84f8bbd2e90c6027d0c1749b5b19a8eb28987ad6cc734
May 20 15:42:42.829646 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:42.829607    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pzwkl" event={"ID":"e006510d-31f4-435c-9001-eaae8ed6b5e5","Type":"ContainerStarted","Data":"56e6b5b3584daefb0fa84f8bbd2e90c6027d0c1749b5b19a8eb28987ad6cc734"}
May 20 15:42:46.847257 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:46.847221    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pzwkl" event={"ID":"e006510d-31f4-435c-9001-eaae8ed6b5e5","Type":"ContainerStarted","Data":"7a9697f76bcb95dc5f6b2024f410d2fd1176174ed2d29125dc595e830144a7b8"}
May 20 15:42:46.864741 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:46.864694    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-pzwkl" podStartSLOduration=1.480786384 podStartE2EDuration="5.864680319s" podCreationTimestamp="2026-05-20 15:42:41 +0000 UTC" firstStartedPulling="2026-05-20 15:42:41.938263733 +0000 UTC m=+670.877105269" lastFinishedPulling="2026-05-20 15:42:46.322157666 +0000 UTC m=+675.260999204" observedRunningTime="2026-05-20 15:42:46.862763593 +0000 UTC m=+675.801605149" watchObservedRunningTime="2026-05-20 15:42:46.864680319 +0000 UTC m=+675.803521876"
May 20 15:42:49.858506 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:49.858470    2568 generic.go:358] "Generic (PLEG): container finished" podID="e006510d-31f4-435c-9001-eaae8ed6b5e5" containerID="7a9697f76bcb95dc5f6b2024f410d2fd1176174ed2d29125dc595e830144a7b8" exitCode=0
May 20 15:42:49.858872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:49.858545    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pzwkl" event={"ID":"e006510d-31f4-435c-9001-eaae8ed6b5e5","Type":"ContainerDied","Data":"7a9697f76bcb95dc5f6b2024f410d2fd1176174ed2d29125dc595e830144a7b8"}
May 20 15:42:50.992565 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:50.992543    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pzwkl"
May 20 15:42:51.091524 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:51.091488    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlrt2\" (UniqueName: \"kubernetes.io/projected/e006510d-31f4-435c-9001-eaae8ed6b5e5-kube-api-access-wlrt2\") pod \"e006510d-31f4-435c-9001-eaae8ed6b5e5\" (UID: \"e006510d-31f4-435c-9001-eaae8ed6b5e5\") "
May 20 15:42:51.093720 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:51.093689    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e006510d-31f4-435c-9001-eaae8ed6b5e5-kube-api-access-wlrt2" (OuterVolumeSpecName: "kube-api-access-wlrt2") pod "e006510d-31f4-435c-9001-eaae8ed6b5e5" (UID: "e006510d-31f4-435c-9001-eaae8ed6b5e5"). InnerVolumeSpecName "kube-api-access-wlrt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:42:51.192655 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:51.192571    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlrt2\" (UniqueName: \"kubernetes.io/projected/e006510d-31f4-435c-9001-eaae8ed6b5e5-kube-api-access-wlrt2\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:42:51.866517 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:51.866484    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pzwkl" event={"ID":"e006510d-31f4-435c-9001-eaae8ed6b5e5","Type":"ContainerDied","Data":"56e6b5b3584daefb0fa84f8bbd2e90c6027d0c1749b5b19a8eb28987ad6cc734"}
May 20 15:42:51.866517 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:51.866518    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56e6b5b3584daefb0fa84f8bbd2e90c6027d0c1749b5b19a8eb28987ad6cc734"
May 20 15:42:51.866517 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:42:51.866492    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pzwkl"
May 20 15:43:04.211336 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.211293    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"]
May 20 15:43:04.211802 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.211682    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e006510d-31f4-435c-9001-eaae8ed6b5e5" containerName="s3-init"
May 20 15:43:04.211802 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.211695    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e006510d-31f4-435c-9001-eaae8ed6b5e5" containerName="s3-init"
May 20 15:43:04.211802 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.211779    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="e006510d-31f4-435c-9001-eaae8ed6b5e5" containerName="s3-init"
May 20 15:43:04.215192 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.215172    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"
May 20 15:43:04.218902 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.218878    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-542af-kube-rbac-proxy-sar-config\""
May 20 15:43:04.219058 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.218904    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 15:43:04.219058 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.219003    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sgm7r\""
May 20 15:43:04.219808 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.219791    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 15:43:04.220270 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.220253    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-542af-predictor-serving-cert\""
May 20 15:43:04.229500 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.229474    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"]
May 20 15:43:04.306935 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.306897    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shcpg\" (UniqueName: \"kubernetes.io/projected/5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3-kube-api-access-shcpg\") pod \"success-200-isvc-542af-predictor-6f849b64fd-lbxwr\" (UID: \"5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3\") " pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"
May 20 15:43:04.307129 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.306946    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-542af-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3-success-200-isvc-542af-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-542af-predictor-6f849b64fd-lbxwr\" (UID: \"5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3\") " pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"
May 20 15:43:04.307129 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.307014    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3-proxy-tls\") pod \"success-200-isvc-542af-predictor-6f849b64fd-lbxwr\" (UID: \"5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3\") " pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"
May 20 15:43:04.408506 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.408474    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-542af-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3-success-200-isvc-542af-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-542af-predictor-6f849b64fd-lbxwr\" (UID: \"5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3\") " pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"
May 20 15:43:04.408692 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.408560    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3-proxy-tls\") pod \"success-200-isvc-542af-predictor-6f849b64fd-lbxwr\" (UID: \"5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3\") " pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"
May 20 15:43:04.408692 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.408621    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-shcpg\" (UniqueName: \"kubernetes.io/projected/5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3-kube-api-access-shcpg\") pod \"success-200-isvc-542af-predictor-6f849b64fd-lbxwr\" (UID: \"5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3\") " pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"
May 20 15:43:04.409301 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.409272    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-542af-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3-success-200-isvc-542af-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-542af-predictor-6f849b64fd-lbxwr\" (UID: \"5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3\") " pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"
May 20 15:43:04.411462 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.411435    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3-proxy-tls\") pod \"success-200-isvc-542af-predictor-6f849b64fd-lbxwr\" (UID: \"5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3\") " pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"
May 20 15:43:04.420109 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.420058    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shcpg\" (UniqueName: \"kubernetes.io/projected/5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3-kube-api-access-shcpg\") pod \"success-200-isvc-542af-predictor-6f849b64fd-lbxwr\" (UID: \"5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3\") " pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"
May 20 15:43:04.525916 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.525835    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"
May 20 15:43:04.614165 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.614127    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"]
May 20 15:43:04.619591 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.619566    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"
May 20 15:43:04.622193 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.622086    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
May 20 15:43:04.622613 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.622430    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
May 20 15:43:04.633138 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.630409    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"]
May 20 15:43:04.703497 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.703467    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"]
May 20 15:43:04.705332 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:43:04.705305    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f54aa33_a2d5_4ac1_8ff3_d4bc9b747ed3.slice/crio-d92fa8db269b0af46f917003651744b27822c57545d9828ef39eeb26d1b588d0 WatchSource:0}: Error finding container d92fa8db269b0af46f917003651744b27822c57545d9828ef39eeb26d1b588d0: Status 404 returned error can't find the container with id d92fa8db269b0af46f917003651744b27822c57545d9828ef39eeb26d1b588d0
May 20 15:43:04.707122 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.707107    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:43:04.712038 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.711972    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15b10f19-0c65-4064-b0d7-7213aa008b0b-proxy-tls\") pod \"isvc-xgboost-graph-predictor-669d8d6456-h8dq4\" (UID: \"15b10f19-0c65-4064-b0d7-7213aa008b0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"
May 20 15:43:04.712120 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.712047    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15b10f19-0c65-4064-b0d7-7213aa008b0b-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-669d8d6456-h8dq4\" (UID: \"15b10f19-0c65-4064-b0d7-7213aa008b0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"
May 20 15:43:04.712360 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.712233    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15b10f19-0c65-4064-b0d7-7213aa008b0b-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-669d8d6456-h8dq4\" (UID: \"15b10f19-0c65-4064-b0d7-7213aa008b0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"
May 20 15:43:04.712360 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.712269    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7tnp\" (UniqueName: \"kubernetes.io/projected/15b10f19-0c65-4064-b0d7-7213aa008b0b-kube-api-access-w7tnp\") pod \"isvc-xgboost-graph-predictor-669d8d6456-h8dq4\" (UID: \"15b10f19-0c65-4064-b0d7-7213aa008b0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"
May 20 15:43:04.735518 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.735496    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"]
May 20 15:43:04.739043 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.739013    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"
May 20 15:43:04.742494 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.742477    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-542af-predictor-serving-cert\""
May 20 15:43:04.742595 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.742495    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-542af-kube-rbac-proxy-sar-config\""
May 20 15:43:04.755947 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.755914    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"]
May 20 15:43:04.813268 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.813235    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-542af-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/957084cc-60ef-4b62-b1e7-57642a9adad7-error-404-isvc-542af-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-542af-predictor-5854dfc889-f25kz\" (UID: \"957084cc-60ef-4b62-b1e7-57642a9adad7\") " pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"
May 20 15:43:04.813268 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.813271    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15b10f19-0c65-4064-b0d7-7213aa008b0b-proxy-tls\") pod \"isvc-xgboost-graph-predictor-669d8d6456-h8dq4\" (UID: \"15b10f19-0c65-4064-b0d7-7213aa008b0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"
May 20 15:43:04.813440 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.813290    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15b10f19-0c65-4064-b0d7-7213aa008b0b-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-669d8d6456-h8dq4\" (UID: \"15b10f19-0c65-4064-b0d7-7213aa008b0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"
May 20 15:43:04.813440 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.813310    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/957084cc-60ef-4b62-b1e7-57642a9adad7-proxy-tls\") pod \"error-404-isvc-542af-predictor-5854dfc889-f25kz\" (UID: \"957084cc-60ef-4b62-b1e7-57642a9adad7\") " pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"
May 20 15:43:04.813440 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.813365    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15b10f19-0c65-4064-b0d7-7213aa008b0b-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-669d8d6456-h8dq4\" (UID: \"15b10f19-0c65-4064-b0d7-7213aa008b0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"
May 20 15:43:04.813440 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.813382    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w7tnp\" (UniqueName: \"kubernetes.io/projected/15b10f19-0c65-4064-b0d7-7213aa008b0b-kube-api-access-w7tnp\") pod \"isvc-xgboost-graph-predictor-669d8d6456-h8dq4\" (UID: \"15b10f19-0c65-4064-b0d7-7213aa008b0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"
May 20 15:43:04.813440 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.813407    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2pgz\" (UniqueName: \"kubernetes.io/projected/957084cc-60ef-4b62-b1e7-57642a9adad7-kube-api-access-c2pgz\") pod \"error-404-isvc-542af-predictor-5854dfc889-f25kz\" (UID: \"957084cc-60ef-4b62-b1e7-57642a9adad7\") " pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"
May 20 15:43:04.813724 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.813704    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15b10f19-0c65-4064-b0d7-7213aa008b0b-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-669d8d6456-h8dq4\" (UID: \"15b10f19-0c65-4064-b0d7-7213aa008b0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"
May 20 15:43:04.814073 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.814048    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15b10f19-0c65-4064-b0d7-7213aa008b0b-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-669d8d6456-h8dq4\" (UID: \"15b10f19-0c65-4064-b0d7-7213aa008b0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"
May 20 15:43:04.815876 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.815857    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15b10f19-0c65-4064-b0d7-7213aa008b0b-proxy-tls\") pod \"isvc-xgboost-graph-predictor-669d8d6456-h8dq4\" (UID: \"15b10f19-0c65-4064-b0d7-7213aa008b0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"
May 20 15:43:04.825122 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.825102    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7tnp\" (UniqueName: \"kubernetes.io/projected/15b10f19-0c65-4064-b0d7-7213aa008b0b-kube-api-access-w7tnp\") pod \"isvc-xgboost-graph-predictor-669d8d6456-h8dq4\" (UID: \"15b10f19-0c65-4064-b0d7-7213aa008b0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"
May 20 15:43:04.910706 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.910673    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr" event={"ID":"5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3","Type":"ContainerStarted","Data":"d92fa8db269b0af46f917003651744b27822c57545d9828ef39eeb26d1b588d0"}
May 20 15:43:04.914154 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.914132    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c2pgz\" (UniqueName: \"kubernetes.io/projected/957084cc-60ef-4b62-b1e7-57642a9adad7-kube-api-access-c2pgz\") pod \"error-404-isvc-542af-predictor-5854dfc889-f25kz\" (UID: \"957084cc-60ef-4b62-b1e7-57642a9adad7\") " pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"
May 20 15:43:04.914288 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.914174    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-542af-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/957084cc-60ef-4b62-b1e7-57642a9adad7-error-404-isvc-542af-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-542af-predictor-5854dfc889-f25kz\" (UID: \"957084cc-60ef-4b62-b1e7-57642a9adad7\") " pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"
May 20 15:43:04.914346 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.914299    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/957084cc-60ef-4b62-b1e7-57642a9adad7-proxy-tls\") pod \"error-404-isvc-542af-predictor-5854dfc889-f25kz\" (UID: \"957084cc-60ef-4b62-b1e7-57642a9adad7\") " pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"
May 20 15:43:04.914773 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.914750    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-542af-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/957084cc-60ef-4b62-b1e7-57642a9adad7-error-404-isvc-542af-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-542af-predictor-5854dfc889-f25kz\" (UID: \"957084cc-60ef-4b62-b1e7-57642a9adad7\") " pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"
May 20 15:43:04.916723 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.916704    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/957084cc-60ef-4b62-b1e7-57642a9adad7-proxy-tls\") pod \"error-404-isvc-542af-predictor-5854dfc889-f25kz\" (UID: \"957084cc-60ef-4b62-b1e7-57642a9adad7\") " pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"
May 20 15:43:04.923421 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.923402    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2pgz\" (UniqueName: \"kubernetes.io/projected/957084cc-60ef-4b62-b1e7-57642a9adad7-kube-api-access-c2pgz\") pod \"error-404-isvc-542af-predictor-5854dfc889-f25kz\" (UID: \"957084cc-60ef-4b62-b1e7-57642a9adad7\") " pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"
May 20 15:43:04.937254 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:04.937236    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"
May 20 15:43:05.052458 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:05.052429    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"
May 20 15:43:05.067079 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:05.067051    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"]
May 20 15:43:05.068361 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:43:05.068332    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b10f19_0c65_4064_b0d7_7213aa008b0b.slice/crio-88ec3bf872796be33a959ca7a45f601093007af070ed6f537bb2df0915e60db1 WatchSource:0}: Error finding container 88ec3bf872796be33a959ca7a45f601093007af070ed6f537bb2df0915e60db1: Status 404 returned error can't find the container with id 88ec3bf872796be33a959ca7a45f601093007af070ed6f537bb2df0915e60db1
May 20 15:43:05.204519 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:05.204490    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"]
May 20 15:43:05.206730 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:43:05.206697    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957084cc_60ef_4b62_b1e7_57642a9adad7.slice/crio-0a3f66c1547e6e66368ce3f531858fea18b77f79da14833c02ed5007050a7bef WatchSource:0}: Error finding container 0a3f66c1547e6e66368ce3f531858fea18b77f79da14833c02ed5007050a7bef: Status 404 returned error can't find the container with id 0a3f66c1547e6e66368ce3f531858fea18b77f79da14833c02ed5007050a7bef
May 20 15:43:05.922059 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:05.921984    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz" event={"ID":"957084cc-60ef-4b62-b1e7-57642a9adad7","Type":"ContainerStarted","Data":"0a3f66c1547e6e66368ce3f531858fea18b77f79da14833c02ed5007050a7bef"}
May 20 15:43:05.927754 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:05.927723    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4" event={"ID":"15b10f19-0c65-4064-b0d7-7213aa008b0b","Type":"ContainerStarted","Data":"88ec3bf872796be33a959ca7a45f601093007af070ed6f537bb2df0915e60db1"}
May 20 15:43:10.957989 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:10.957904    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4" event={"ID":"15b10f19-0c65-4064-b0d7-7213aa008b0b","Type":"ContainerStarted","Data":"fa964618baec4f9db6d5e6ab764d99fe5c555648c62bca5564fe7334fb13c4b4"}
May 20 15:43:15.979235 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:15.979195    2568 generic.go:358] "Generic (PLEG): container finished" podID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerID="fa964618baec4f9db6d5e6ab764d99fe5c555648c62bca5564fe7334fb13c4b4" exitCode=0
May 20 15:43:15.979693 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:15.979251    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4" event={"ID":"15b10f19-0c65-4064-b0d7-7213aa008b0b","Type":"ContainerDied","Data":"fa964618baec4f9db6d5e6ab764d99fe5c555648c62bca5564fe7334fb13c4b4"}
May 20 15:43:21.005214 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:21.005140    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr" event={"ID":"5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3","Type":"ContainerStarted","Data":"c86f8f6e6f13ede72e5228d44ecedc28ece6489a514e1b9300a87308b621609d"}
May 20 15:43:21.012363 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:21.012286    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz" event={"ID":"957084cc-60ef-4b62-b1e7-57642a9adad7","Type":"ContainerStarted","Data":"0e8eb3d2a416aae7b48df4f801e42b11c1442923b92bc710105f1a1b2ba5b188"}
May 20 15:43:25.033806 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:25.033764    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz" event={"ID":"957084cc-60ef-4b62-b1e7-57642a9adad7","Type":"ContainerStarted","Data":"cd285dd20daa9f7a87d369a69d132e02c2d32377f12ee34602f8504001a7b4fe"}
May 20 15:43:26.039344 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:26.039301    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr" event={"ID":"5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3","Type":"ContainerStarted","Data":"1a6b38bcb338616a7852cc7ff3348edaca955e7a6d2f8d9b1371ed37153c5642"}
May 20 15:43:26.039901 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:26.039414    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"
May 20 15:43:26.039901 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:26.039786    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"
May 20 15:43:26.039901 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:26.039812    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"
May 20 15:43:26.041398 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:26.041367    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz" podUID="957084cc-60ef-4b62-b1e7-57642a9adad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 20 15:43:26.066866 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:26.066799    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr" podStartSLOduration=1.819968598 podStartE2EDuration="22.066784959s" podCreationTimestamp="2026-05-20 15:43:04 +0000 UTC" firstStartedPulling="2026-05-20 15:43:04.707229548 +0000 UTC m=+693.646071082" lastFinishedPulling="2026-05-20 15:43:24.954045894 +0000 UTC m=+713.892887443" observedRunningTime="2026-05-20 15:43:26.064061992 +0000 UTC m=+715.002903549" watchObservedRunningTime="2026-05-20 15:43:26.066784959 +0000 UTC m=+715.005626518"
May 20 15:43:26.089716 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:26.089668    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz" podStartSLOduration=2.360899393 podStartE2EDuration="22.089651695s" podCreationTimestamp="2026-05-20 15:43:04 +0000 UTC" firstStartedPulling="2026-05-20 15:43:05.208937559 +0000 UTC m=+694.147779095" lastFinishedPulling="2026-05-20 15:43:24.937689847 +0000 UTC m=+713.876531397" observedRunningTime="2026-05-20 15:43:26.089289035 +0000 UTC m=+715.028130592" watchObservedRunningTime="2026-05-20 15:43:26.089651695 +0000 UTC m=+715.028493253"
May 20 15:43:27.043522 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:27.043487    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"
May 20 15:43:27.043943 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:27.043538    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz" podUID="957084cc-60ef-4b62-b1e7-57642a9adad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 20 15:43:27.045120 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:27.045076    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr" podUID="5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 20 15:43:28.047396 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:28.047350    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz" podUID="957084cc-60ef-4b62-b1e7-57642a9adad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 20 15:43:28.047840 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:28.047401    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr" podUID="5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 20 15:43:33.052927 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:33.052892    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"
May 20 15:43:33.053434 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:33.053175    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"
May 20 15:43:33.053434 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:33.053370    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr" podUID="5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 20 15:43:33.053812 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:33.053792    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz" podUID="957084cc-60ef-4b62-b1e7-57642a9adad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 20 15:43:41.098203 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:41.098168    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4" event={"ID":"15b10f19-0c65-4064-b0d7-7213aa008b0b","Type":"ContainerStarted","Data":"623f2ddcf89e2da0fe028c6fdcae57b90d018f693350d4dda92b352852c7b575"}
May 20 15:43:41.098635 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:41.098212    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4" event={"ID":"15b10f19-0c65-4064-b0d7-7213aa008b0b","Type":"ContainerStarted","Data":"81c2b60d17c0f3a8c089f83410bb30251ff31756e0d3559fdcb79bde59a8269f"}
May 20 15:43:41.098635 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:41.098419    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"
May 20 15:43:41.119555 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:41.119482    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4" podStartSLOduration=1.7150626469999999 podStartE2EDuration="37.119460009s" podCreationTimestamp="2026-05-20 15:43:04 +0000 UTC" firstStartedPulling="2026-05-20 15:43:05.070195293 +0000 UTC m=+694.009036828" lastFinishedPulling="2026-05-20 15:43:40.474592636 +0000 UTC m=+729.413434190" observedRunningTime="2026-05-20 15:43:41.118729178 +0000 UTC m=+730.057570734" watchObservedRunningTime="2026-05-20 15:43:41.119460009 +0000 UTC m=+730.058301564"
May 20 15:43:42.102119 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:42.102084    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"
May 20 15:43:42.103247 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:42.103221    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4" podUID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 20 15:43:43.054433 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:43.054385    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz" podUID="957084cc-60ef-4b62-b1e7-57642a9adad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 20 15:43:43.054620 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:43.054385    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr" podUID="5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 20 15:43:43.105112 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:43.105065    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4" podUID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 20 15:43:48.110937 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:48.110905    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"
May 20 15:43:48.111520 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:48.111489    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4" podUID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 20 15:43:53.053540 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:53.053497    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr" podUID="5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 20 15:43:53.053960 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:53.053816    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz" podUID="957084cc-60ef-4b62-b1e7-57642a9adad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 20 15:43:58.112341 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:43:58.112297    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4" podUID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 20 15:44:03.053383 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:03.053340    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr" podUID="5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 20 15:44:03.053802 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:03.053779    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz" podUID="957084cc-60ef-4b62-b1e7-57642a9adad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 20 15:44:08.112459 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:08.112412    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4" podUID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 20 15:44:13.054557 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:13.054523    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"
May 20 15:44:13.055087 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:13.054688    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"
May 20 15:44:18.111551 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:18.111510    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4" podUID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 20 15:44:24.289429 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:24.289395    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk"]
May 20 15:44:24.291648 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:24.291631    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk"
May 20 15:44:24.294239 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:24.294216    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-542af-serving-cert\""
May 20 15:44:24.294360 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:24.294218    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-542af-kube-rbac-proxy-sar-config\""
May 20 15:44:24.301218 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:24.301194    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk"]
May 20 15:44:24.348035 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:24.347980    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b90118ab-d260-43ff-8b74-c492f22c2c6c-proxy-tls\") pod \"switch-graph-542af-d57596c4d-5w6hk\" (UID: \"b90118ab-d260-43ff-8b74-c492f22c2c6c\") " pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk"
May 20 15:44:24.348218 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:24.348078    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b90118ab-d260-43ff-8b74-c492f22c2c6c-openshift-service-ca-bundle\") pod \"switch-graph-542af-d57596c4d-5w6hk\" (UID: \"b90118ab-d260-43ff-8b74-c492f22c2c6c\") " pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk"
May 20 15:44:24.449202 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:24.449167    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b90118ab-d260-43ff-8b74-c492f22c2c6c-openshift-service-ca-bundle\") pod \"switch-graph-542af-d57596c4d-5w6hk\" (UID: \"b90118ab-d260-43ff-8b74-c492f22c2c6c\") " pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk"
May 20 15:44:24.449409 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:24.449289    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b90118ab-d260-43ff-8b74-c492f22c2c6c-proxy-tls\") pod \"switch-graph-542af-d57596c4d-5w6hk\" (UID: \"b90118ab-d260-43ff-8b74-c492f22c2c6c\") " pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk"
May 20 15:44:24.449483 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:44:24.449433    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-542af-serving-cert: secret "switch-graph-542af-serving-cert" not found
May 20 15:44:24.449537 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:44:24.449521    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b90118ab-d260-43ff-8b74-c492f22c2c6c-proxy-tls podName:b90118ab-d260-43ff-8b74-c492f22c2c6c nodeName:}" failed. No retries permitted until 2026-05-20 15:44:24.949498669 +0000 UTC m=+773.888340220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b90118ab-d260-43ff-8b74-c492f22c2c6c-proxy-tls") pod "switch-graph-542af-d57596c4d-5w6hk" (UID: "b90118ab-d260-43ff-8b74-c492f22c2c6c") : secret "switch-graph-542af-serving-cert" not found
May 20 15:44:24.449796 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:24.449771    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b90118ab-d260-43ff-8b74-c492f22c2c6c-openshift-service-ca-bundle\") pod \"switch-graph-542af-d57596c4d-5w6hk\" (UID: \"b90118ab-d260-43ff-8b74-c492f22c2c6c\") " pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk"
May 20 15:44:24.953896 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:24.953864    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b90118ab-d260-43ff-8b74-c492f22c2c6c-proxy-tls\") pod \"switch-graph-542af-d57596c4d-5w6hk\" (UID: \"b90118ab-d260-43ff-8b74-c492f22c2c6c\") " pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk"
May 20 15:44:24.956437 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:24.956419    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b90118ab-d260-43ff-8b74-c492f22c2c6c-proxy-tls\") pod \"switch-graph-542af-d57596c4d-5w6hk\" (UID: \"b90118ab-d260-43ff-8b74-c492f22c2c6c\") " pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk"
May 20 15:44:25.203256 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:25.203218    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk"
May 20 15:44:25.332409 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:25.332384    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk"]
May 20 15:44:25.334751 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:44:25.334722    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb90118ab_d260_43ff_8b74_c492f22c2c6c.slice/crio-77588b3bd204e8a81662cbaea86272d76002c676f1f5d2cbdcb8b20c8c686224 WatchSource:0}: Error finding container 77588b3bd204e8a81662cbaea86272d76002c676f1f5d2cbdcb8b20c8c686224: Status 404 returned error can't find the container with id 77588b3bd204e8a81662cbaea86272d76002c676f1f5d2cbdcb8b20c8c686224
May 20 15:44:26.259906 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:26.259871    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk" event={"ID":"b90118ab-d260-43ff-8b74-c492f22c2c6c","Type":"ContainerStarted","Data":"77588b3bd204e8a81662cbaea86272d76002c676f1f5d2cbdcb8b20c8c686224"}
May 20 15:44:28.111447 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:28.111407    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4" podUID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 20 15:44:28.270038 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:28.269981    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk" event={"ID":"b90118ab-d260-43ff-8b74-c492f22c2c6c","Type":"ContainerStarted","Data":"be53ca626d6abdb9a1fff0af8b54976e580058d3b53b69dc32c56177edcb8e95"}
May 20 15:44:28.270223 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:28.270091    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk"
May 20 15:44:28.306971 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:28.306920    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk" podStartSLOduration=1.8792535259999998 podStartE2EDuration="4.306905811s" podCreationTimestamp="2026-05-20 15:44:24 +0000 UTC" firstStartedPulling="2026-05-20 15:44:25.336411242 +0000 UTC m=+774.275252777" lastFinishedPulling="2026-05-20 15:44:27.764063527 +0000 UTC m=+776.702905062" observedRunningTime="2026-05-20 15:44:28.3062128 +0000 UTC m=+777.245054357" watchObservedRunningTime="2026-05-20 15:44:28.306905811 +0000 UTC m=+777.245747368"
May 20 15:44:34.280850 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:34.280818    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk"
May 20 15:44:38.111652 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:38.111606    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4" podUID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 20 15:44:38.505860 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:38.502333    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk"]
May 20 15:44:38.505860 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:38.502706    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk" podUID="b90118ab-d260-43ff-8b74-c492f22c2c6c" containerName="switch-graph-542af" containerID="cri-o://be53ca626d6abdb9a1fff0af8b54976e580058d3b53b69dc32c56177edcb8e95" gracePeriod=30
May 20 15:44:38.720529 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:38.720491    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"]
May 20 15:44:38.720814 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:38.720788    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr" podUID="5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3" containerName="kserve-container" containerID="cri-o://c86f8f6e6f13ede72e5228d44ecedc28ece6489a514e1b9300a87308b621609d" gracePeriod=30
May 20 15:44:38.720879 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:38.720828    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr" podUID="5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3" containerName="kube-rbac-proxy" containerID="cri-o://1a6b38bcb338616a7852cc7ff3348edaca955e7a6d2f8d9b1371ed37153c5642" gracePeriod=30
May 20 15:44:38.883970 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:38.883936    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"]
May 20 15:44:38.884403 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:38.884341    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz" podUID="957084cc-60ef-4b62-b1e7-57642a9adad7" containerName="kserve-container" containerID="cri-o://0e8eb3d2a416aae7b48df4f801e42b11c1442923b92bc710105f1a1b2ba5b188" gracePeriod=30
May 20 15:44:38.884569 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:38.884370    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz" podUID="957084cc-60ef-4b62-b1e7-57642a9adad7" containerName="kube-rbac-proxy" containerID="cri-o://cd285dd20daa9f7a87d369a69d132e02c2d32377f12ee34602f8504001a7b4fe" gracePeriod=30
May 20 15:44:38.912204 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:38.912174    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"]
May 20 15:44:38.914679 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:38.914664    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"
May 20 15:44:38.918706 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:38.918677    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d4d66-kube-rbac-proxy-sar-config\""
May 20 15:44:38.922462 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:38.922437    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"]
May 20 15:44:38.925993 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:38.925971    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d4d66-predictor-serving-cert\""
May 20 15:44:38.980722 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:38.980691    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a538143-ec1a-49c3-9204-e35cf7adbd2e-proxy-tls\") pod \"success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg\" (UID: \"1a538143-ec1a-49c3-9204-e35cf7adbd2e\") " pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"
May 20 15:44:38.980722 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:38.980730    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8vf5\" (UniqueName: \"kubernetes.io/projected/1a538143-ec1a-49c3-9204-e35cf7adbd2e-kube-api-access-s8vf5\") pod \"success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg\" (UID: \"1a538143-ec1a-49c3-9204-e35cf7adbd2e\") " pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"
May 20 15:44:38.980966 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:38.980755    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-d4d66-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a538143-ec1a-49c3-9204-e35cf7adbd2e-success-200-isvc-d4d66-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg\" (UID: \"1a538143-ec1a-49c3-9204-e35cf7adbd2e\") " pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"
May 20 15:44:39.082151 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.082113    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a538143-ec1a-49c3-9204-e35cf7adbd2e-proxy-tls\") pod \"success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg\" (UID: \"1a538143-ec1a-49c3-9204-e35cf7adbd2e\") " pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"
May 20 15:44:39.082151 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.082155    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s8vf5\" (UniqueName: \"kubernetes.io/projected/1a538143-ec1a-49c3-9204-e35cf7adbd2e-kube-api-access-s8vf5\") pod \"success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg\" (UID: \"1a538143-ec1a-49c3-9204-e35cf7adbd2e\") " pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"
May 20 15:44:39.082394 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.082177    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-d4d66-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a538143-ec1a-49c3-9204-e35cf7adbd2e-success-200-isvc-d4d66-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg\" (UID: \"1a538143-ec1a-49c3-9204-e35cf7adbd2e\") " pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"
May 20 15:44:39.082951 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.082929    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-d4d66-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a538143-ec1a-49c3-9204-e35cf7adbd2e-success-200-isvc-d4d66-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg\" (UID: \"1a538143-ec1a-49c3-9204-e35cf7adbd2e\") " pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"
May 20 15:44:39.084864 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.084840    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a538143-ec1a-49c3-9204-e35cf7adbd2e-proxy-tls\") pod \"success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg\" (UID: \"1a538143-ec1a-49c3-9204-e35cf7adbd2e\") " pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"
May 20 15:44:39.088368 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.088285    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"]
May 20 15:44:39.090931 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.090913    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"
May 20 15:44:39.098381 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.098359    2568 status_manager.go:895] "Failed to get status for pod" podUID="4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5" err="pods \"error-404-isvc-d4d66-predictor-867b8b47f7-js4z5\" is forbidden: User \"system:node:ip-10-0-129-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-129-117.ec2.internal' and this object"
May 20 15:44:39.098469 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:44:39.098434    2568 reflector.go:200] "Failed to watch" err="failed to list *v1.Secret: secrets \"error-404-isvc-d4d66-predictor-serving-cert\" is forbidden: User \"system:node:ip-10-0-129-117.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-129-117.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d4d66-predictor-serving-cert\"" type="*v1.Secret"
May 20 15:44:39.098469 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:44:39.098444    2568 reflector.go:200] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"error-404-isvc-d4d66-kube-rbac-proxy-sar-config\" is forbidden: User \"system:node:ip-10-0-129-117.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-129-117.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d4d66-kube-rbac-proxy-sar-config\"" type="*v1.ConfigMap"
May 20 15:44:39.110435 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.110413    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"]
May 20 15:44:39.124394 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.124367    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8vf5\" (UniqueName: \"kubernetes.io/projected/1a538143-ec1a-49c3-9204-e35cf7adbd2e-kube-api-access-s8vf5\") pod \"success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg\" (UID: \"1a538143-ec1a-49c3-9204-e35cf7adbd2e\") " pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"
May 20 15:44:39.183448 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.183409    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvbz7\" (UniqueName: \"kubernetes.io/projected/4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b-kube-api-access-jvbz7\") pod \"error-404-isvc-d4d66-predictor-867b8b47f7-js4z5\" (UID: \"4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b\") " pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"
May 20 15:44:39.183633 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.183478    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-d4d66-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b-error-404-isvc-d4d66-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d4d66-predictor-867b8b47f7-js4z5\" (UID: \"4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b\") " pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"
May 20 15:44:39.183633 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.183548    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b-proxy-tls\") pod \"error-404-isvc-d4d66-predictor-867b8b47f7-js4z5\" (UID: \"4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b\") " pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"
May 20 15:44:39.226385 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.226355    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"
May 20 15:44:39.280835 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.280794    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk" podUID="b90118ab-d260-43ff-8b74-c492f22c2c6c" containerName="switch-graph-542af" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:44:39.284443 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.284304    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-d4d66-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b-error-404-isvc-d4d66-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d4d66-predictor-867b8b47f7-js4z5\" (UID: \"4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b\") " pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"
May 20 15:44:39.284443 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.284434    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b-proxy-tls\") pod \"error-404-isvc-d4d66-predictor-867b8b47f7-js4z5\" (UID: \"4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b\") " pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"
May 20 15:44:39.284631 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.284477    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jvbz7\" (UniqueName: \"kubernetes.io/projected/4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b-kube-api-access-jvbz7\") pod \"error-404-isvc-d4d66-predictor-867b8b47f7-js4z5\" (UID: \"4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b\") " pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"
May 20 15:44:39.309729 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.309679    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvbz7\" (UniqueName: \"kubernetes.io/projected/4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b-kube-api-access-jvbz7\") pod \"error-404-isvc-d4d66-predictor-867b8b47f7-js4z5\" (UID: \"4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b\") " pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"
May 20 15:44:39.310841 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.310815    2568 generic.go:358] "Generic (PLEG): container finished" podID="5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3" containerID="1a6b38bcb338616a7852cc7ff3348edaca955e7a6d2f8d9b1371ed37153c5642" exitCode=2
May 20 15:44:39.310977 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.310886    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr" event={"ID":"5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3","Type":"ContainerDied","Data":"1a6b38bcb338616a7852cc7ff3348edaca955e7a6d2f8d9b1371ed37153c5642"}
May 20 15:44:39.312414 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.312386    2568 generic.go:358] "Generic (PLEG): container finished" podID="957084cc-60ef-4b62-b1e7-57642a9adad7" containerID="cd285dd20daa9f7a87d369a69d132e02c2d32377f12ee34602f8504001a7b4fe" exitCode=2
May 20 15:44:39.312560 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.312542    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz" event={"ID":"957084cc-60ef-4b62-b1e7-57642a9adad7","Type":"ContainerDied","Data":"cd285dd20daa9f7a87d369a69d132e02c2d32377f12ee34602f8504001a7b4fe"}
May 20 15:44:39.393700 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:39.393673    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"]
May 20 15:44:39.394689 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:44:39.394664    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a538143_ec1a_49c3_9204_e35cf7adbd2e.slice/crio-c1249ad428d4a1d204ce2627e94c0c068d51912306a126f72bf5bfab7824569f WatchSource:0}: Error finding container c1249ad428d4a1d204ce2627e94c0c068d51912306a126f72bf5bfab7824569f: Status 404 returned error can't find the container with id c1249ad428d4a1d204ce2627e94c0c068d51912306a126f72bf5bfab7824569f
May 20 15:44:40.134278 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:40.134245    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d4d66-predictor-serving-cert\""
May 20 15:44:40.137611 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:40.137590    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b-proxy-tls\") pod \"error-404-isvc-d4d66-predictor-867b8b47f7-js4z5\" (UID: \"4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b\") " pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"
May 20 15:44:40.190861 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:40.190828    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d4d66-kube-rbac-proxy-sar-config\""
May 20 15:44:40.195449 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:40.195430    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-d4d66-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b-error-404-isvc-d4d66-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d4d66-predictor-867b8b47f7-js4z5\" (UID: \"4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b\") " pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"
May 20 15:44:40.302361 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:40.302318    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"
May 20 15:44:40.318802 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:40.318656    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg" event={"ID":"1a538143-ec1a-49c3-9204-e35cf7adbd2e","Type":"ContainerStarted","Data":"71831e30ba3629096a4bf0f9630318884c003e44988600665015f88b87e3cc86"}
May 20 15:44:40.318802 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:40.318712    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg" event={"ID":"1a538143-ec1a-49c3-9204-e35cf7adbd2e","Type":"ContainerStarted","Data":"efdfbb1c5bae91bb24aae5b5868374fd18cd8b110a0d828936f0e9a1c11ca75c"}
May 20 15:44:40.318802 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:40.318728    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg" event={"ID":"1a538143-ec1a-49c3-9204-e35cf7adbd2e","Type":"ContainerStarted","Data":"c1249ad428d4a1d204ce2627e94c0c068d51912306a126f72bf5bfab7824569f"}
May 20 15:44:40.319190 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:40.319010    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"
May 20 15:44:40.319190 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:40.319081    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"
May 20 15:44:40.320515 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:40.320485    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg" podUID="1a538143-ec1a-49c3-9204-e35cf7adbd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 20 15:44:40.385971 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:40.385851    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg" podStartSLOduration=2.385833966 podStartE2EDuration="2.385833966s" podCreationTimestamp="2026-05-20 15:44:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:44:40.382823854 +0000 UTC m=+789.321665413" watchObservedRunningTime="2026-05-20 15:44:40.385833966 +0000 UTC m=+789.324675513"
May 20 15:44:40.460500 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:40.460472    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"]
May 20 15:44:40.463318 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:44:40.463278    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa54823_53e0_4d1e_aafe_9c7f98cc3c0b.slice/crio-b1255f5852df0557555b34f02a6cc4a7f155af2831a95acbbc20241609ff094e WatchSource:0}: Error finding container b1255f5852df0557555b34f02a6cc4a7f155af2831a95acbbc20241609ff094e: Status 404 returned error can't find the container with id b1255f5852df0557555b34f02a6cc4a7f155af2831a95acbbc20241609ff094e
May 20 15:44:41.324727 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:41.324690    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5" event={"ID":"4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b","Type":"ContainerStarted","Data":"1bb6886bb8e627fbd06a2116f8278a7586f991d6b4e80df3bcd234e1738369ec"}
May 20 15:44:41.324727 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:41.324733    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5" event={"ID":"4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b","Type":"ContainerStarted","Data":"cf92198c9d9f5e9e63795e4e62194f097be5b09cec181c81a8521944b6b294c6"}
May 20 15:44:41.325244 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:41.324746    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5" event={"ID":"4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b","Type":"ContainerStarted","Data":"b1255f5852df0557555b34f02a6cc4a7f155af2831a95acbbc20241609ff094e"}
May 20 15:44:41.325244 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:41.324773    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"
May 20 15:44:41.325244 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:41.325003    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg" podUID="1a538143-ec1a-49c3-9204-e35cf7adbd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 20 15:44:41.371815 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:41.371755    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5" podStartSLOduration=2.371732798 podStartE2EDuration="2.371732798s" podCreationTimestamp="2026-05-20 15:44:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:44:41.370460898 +0000 UTC m=+790.309302477" watchObservedRunningTime="2026-05-20 15:44:41.371732798 +0000 UTC m=+790.310574357"
May 20 15:44:42.330118 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.330038    2568 generic.go:358] "Generic (PLEG): container finished" podID="5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3" containerID="c86f8f6e6f13ede72e5228d44ecedc28ece6489a514e1b9300a87308b621609d" exitCode=0
May 20 15:44:42.330500 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.330121    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr" event={"ID":"5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3","Type":"ContainerDied","Data":"c86f8f6e6f13ede72e5228d44ecedc28ece6489a514e1b9300a87308b621609d"}
May 20 15:44:42.330500 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.330153    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr" event={"ID":"5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3","Type":"ContainerDied","Data":"d92fa8db269b0af46f917003651744b27822c57545d9828ef39eeb26d1b588d0"}
May 20 15:44:42.330500 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.330163    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d92fa8db269b0af46f917003651744b27822c57545d9828ef39eeb26d1b588d0"
May 20 15:44:42.331937 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.331911    2568 generic.go:358] "Generic (PLEG): container finished" podID="957084cc-60ef-4b62-b1e7-57642a9adad7" containerID="0e8eb3d2a416aae7b48df4f801e42b11c1442923b92bc710105f1a1b2ba5b188" exitCode=0
May 20 15:44:42.332142 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.331947    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz" event={"ID":"957084cc-60ef-4b62-b1e7-57642a9adad7","Type":"ContainerDied","Data":"0e8eb3d2a416aae7b48df4f801e42b11c1442923b92bc710105f1a1b2ba5b188"}
May 20 15:44:42.332419 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.332397    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"
May 20 15:44:42.333233 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.333215    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"
May 20 15:44:42.334174 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.334146    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5" podUID="4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 20 15:44:42.413173 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.413141    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-542af-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3-success-200-isvc-542af-kube-rbac-proxy-sar-config\") pod \"5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3\" (UID: \"5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3\") "
May 20 15:44:42.413173 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.413179    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3-proxy-tls\") pod \"5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3\" (UID: \"5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3\") "
May 20 15:44:42.413425 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.413216    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shcpg\" (UniqueName: \"kubernetes.io/projected/5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3-kube-api-access-shcpg\") pod \"5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3\" (UID: \"5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3\") "
May 20 15:44:42.413613 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.413491    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3-success-200-isvc-542af-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-542af-kube-rbac-proxy-sar-config") pod "5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3" (UID: "5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3"). InnerVolumeSpecName "success-200-isvc-542af-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:44:42.415633 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.415610    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3" (UID: "5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:44:42.415722 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.415697    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3-kube-api-access-shcpg" (OuterVolumeSpecName: "kube-api-access-shcpg") pod "5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3" (UID: "5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3"). InnerVolumeSpecName "kube-api-access-shcpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:44:42.430804 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.430784    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"
May 20 15:44:42.514178 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.514142    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/957084cc-60ef-4b62-b1e7-57642a9adad7-proxy-tls\") pod \"957084cc-60ef-4b62-b1e7-57642a9adad7\" (UID: \"957084cc-60ef-4b62-b1e7-57642a9adad7\") "
May 20 15:44:42.514178 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.514188    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-542af-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/957084cc-60ef-4b62-b1e7-57642a9adad7-error-404-isvc-542af-kube-rbac-proxy-sar-config\") pod \"957084cc-60ef-4b62-b1e7-57642a9adad7\" (UID: \"957084cc-60ef-4b62-b1e7-57642a9adad7\") "
May 20 15:44:42.514497 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.514307    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2pgz\" (UniqueName: \"kubernetes.io/projected/957084cc-60ef-4b62-b1e7-57642a9adad7-kube-api-access-c2pgz\") pod \"957084cc-60ef-4b62-b1e7-57642a9adad7\" (UID: \"957084cc-60ef-4b62-b1e7-57642a9adad7\") "
May 20 15:44:42.514555 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.514505    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957084cc-60ef-4b62-b1e7-57642a9adad7-error-404-isvc-542af-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-542af-kube-rbac-proxy-sar-config") pod "957084cc-60ef-4b62-b1e7-57642a9adad7" (UID: "957084cc-60ef-4b62-b1e7-57642a9adad7"). InnerVolumeSpecName "error-404-isvc-542af-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:44:42.514617 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.514602    2568 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-542af-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3-success-200-isvc-542af-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:44:42.514617 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.514614    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:44:42.514680 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.514624    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shcpg\" (UniqueName: \"kubernetes.io/projected/5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3-kube-api-access-shcpg\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:44:42.514680 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.514634    2568 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-542af-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/957084cc-60ef-4b62-b1e7-57642a9adad7-error-404-isvc-542af-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:44:42.516567 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.516537    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/957084cc-60ef-4b62-b1e7-57642a9adad7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "957084cc-60ef-4b62-b1e7-57642a9adad7" (UID: "957084cc-60ef-4b62-b1e7-57642a9adad7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:44:42.516567 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.516548    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957084cc-60ef-4b62-b1e7-57642a9adad7-kube-api-access-c2pgz" (OuterVolumeSpecName: "kube-api-access-c2pgz") pod "957084cc-60ef-4b62-b1e7-57642a9adad7" (UID: "957084cc-60ef-4b62-b1e7-57642a9adad7"). InnerVolumeSpecName "kube-api-access-c2pgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:44:42.615283 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.615198    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/957084cc-60ef-4b62-b1e7-57642a9adad7-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:44:42.615283 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:42.615233    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2pgz\" (UniqueName: \"kubernetes.io/projected/957084cc-60ef-4b62-b1e7-57642a9adad7-kube-api-access-c2pgz\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:44:43.336998 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:43.336916    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"
May 20 15:44:43.336998 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:43.336941    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"
May 20 15:44:43.337481 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:43.336913    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz" event={"ID":"957084cc-60ef-4b62-b1e7-57642a9adad7","Type":"ContainerDied","Data":"0a3f66c1547e6e66368ce3f531858fea18b77f79da14833c02ed5007050a7bef"}
May 20 15:44:43.337481 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:43.337062    2568 scope.go:117] "RemoveContainer" containerID="cd285dd20daa9f7a87d369a69d132e02c2d32377f12ee34602f8504001a7b4fe"
May 20 15:44:43.337979 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:43.337727    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5" podUID="4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 20 15:44:43.346430 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:43.346406    2568 scope.go:117] "RemoveContainer" containerID="0e8eb3d2a416aae7b48df4f801e42b11c1442923b92bc710105f1a1b2ba5b188"
May 20 15:44:43.364872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:43.364844    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"]
May 20 15:44:43.370602 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:43.370578    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-542af-predictor-6f849b64fd-lbxwr"]
May 20 15:44:43.383763 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:43.383730    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"]
May 20 15:44:43.385375 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:43.385353    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-542af-predictor-5854dfc889-f25kz"]
May 20 15:44:43.644940 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:43.644861    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3" path="/var/lib/kubelet/pods/5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3/volumes"
May 20 15:44:43.645343 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:43.645328    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="957084cc-60ef-4b62-b1e7-57642a9adad7" path="/var/lib/kubelet/pods/957084cc-60ef-4b62-b1e7-57642a9adad7/volumes"
May 20 15:44:44.279077 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:44.279009    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk" podUID="b90118ab-d260-43ff-8b74-c492f22c2c6c" containerName="switch-graph-542af" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:44:46.329785 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:46.329751    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"
May 20 15:44:46.330426 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:46.330396    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg" podUID="1a538143-ec1a-49c3-9204-e35cf7adbd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 20 15:44:48.113008 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:48.112976    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"
May 20 15:44:48.341565 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:48.341537    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"
May 20 15:44:48.342084 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:48.342057    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5" podUID="4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 20 15:44:49.279954 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:49.279858    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk" podUID="b90118ab-d260-43ff-8b74-c492f22c2c6c" containerName="switch-graph-542af" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:44:49.280354 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:49.279968    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk"
May 20 15:44:54.279331 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:54.279283    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk" podUID="b90118ab-d260-43ff-8b74-c492f22c2c6c" containerName="switch-graph-542af" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:44:56.331251 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:56.331213    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg" podUID="1a538143-ec1a-49c3-9204-e35cf7adbd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 20 15:44:58.342747 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:58.342707    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5" podUID="4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 20 15:44:59.278767 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:44:59.278722    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk" podUID="b90118ab-d260-43ff-8b74-c492f22c2c6c" containerName="switch-graph-542af" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:45:04.279618 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.279570    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk" podUID="b90118ab-d260-43ff-8b74-c492f22c2c6c" containerName="switch-graph-542af" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:45:04.298771 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.298735    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp"]
May 20 15:45:04.299350 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.299334    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3" containerName="kserve-container"
May 20 15:45:04.299441 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.299354    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3" containerName="kserve-container"
May 20 15:45:04.299441 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.299366    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3" containerName="kube-rbac-proxy"
May 20 15:45:04.299441 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.299376    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3" containerName="kube-rbac-proxy"
May 20 15:45:04.299441 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.299399    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="957084cc-60ef-4b62-b1e7-57642a9adad7" containerName="kserve-container"
May 20 15:45:04.299441 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.299408    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="957084cc-60ef-4b62-b1e7-57642a9adad7" containerName="kserve-container"
May 20 15:45:04.299441 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.299419    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="957084cc-60ef-4b62-b1e7-57642a9adad7" containerName="kube-rbac-proxy"
May 20 15:45:04.299441 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.299427    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="957084cc-60ef-4b62-b1e7-57642a9adad7" containerName="kube-rbac-proxy"
May 20 15:45:04.299644 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.299523    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="957084cc-60ef-4b62-b1e7-57642a9adad7" containerName="kserve-container"
May 20 15:45:04.299644 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.299537    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3" containerName="kube-rbac-proxy"
May 20 15:45:04.299644 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.299548    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="5f54aa33-a2d5-4ac1-8ff3-d4bc9b747ed3" containerName="kserve-container"
May 20 15:45:04.299644 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.299561    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="957084cc-60ef-4b62-b1e7-57642a9adad7" containerName="kube-rbac-proxy"
May 20 15:45:04.302986 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.302966    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp"
May 20 15:45:04.307881 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.307860    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
May 20 15:45:04.308004 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.307864    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
May 20 15:45:04.309125 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.309080    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb774bd5-e22c-4742-9d65-794519951feb-openshift-service-ca-bundle\") pod \"model-chainer-6b9459f84c-nwvpp\" (UID: \"fb774bd5-e22c-4742-9d65-794519951feb\") " pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp"
May 20 15:45:04.309245 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.309167    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb774bd5-e22c-4742-9d65-794519951feb-proxy-tls\") pod \"model-chainer-6b9459f84c-nwvpp\" (UID: \"fb774bd5-e22c-4742-9d65-794519951feb\") " pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp"
May 20 15:45:04.320115 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.320088    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp"]
May 20 15:45:04.409890 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.409857    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb774bd5-e22c-4742-9d65-794519951feb-openshift-service-ca-bundle\") pod \"model-chainer-6b9459f84c-nwvpp\" (UID: \"fb774bd5-e22c-4742-9d65-794519951feb\") " pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp"
May 20 15:45:04.410062 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.409905    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb774bd5-e22c-4742-9d65-794519951feb-proxy-tls\") pod \"model-chainer-6b9459f84c-nwvpp\" (UID: \"fb774bd5-e22c-4742-9d65-794519951feb\") " pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp"
May 20 15:45:04.410062 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:45:04.410044    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
May 20 15:45:04.410141 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:45:04.410100    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb774bd5-e22c-4742-9d65-794519951feb-proxy-tls podName:fb774bd5-e22c-4742-9d65-794519951feb nodeName:}" failed. No retries permitted until 2026-05-20 15:45:04.910083664 +0000 UTC m=+813.848925200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fb774bd5-e22c-4742-9d65-794519951feb-proxy-tls") pod "model-chainer-6b9459f84c-nwvpp" (UID: "fb774bd5-e22c-4742-9d65-794519951feb") : secret "model-chainer-serving-cert" not found
May 20 15:45:04.410530 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.410511    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb774bd5-e22c-4742-9d65-794519951feb-openshift-service-ca-bundle\") pod \"model-chainer-6b9459f84c-nwvpp\" (UID: \"fb774bd5-e22c-4742-9d65-794519951feb\") " pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp"
May 20 15:45:04.914627 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.914587    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb774bd5-e22c-4742-9d65-794519951feb-proxy-tls\") pod \"model-chainer-6b9459f84c-nwvpp\" (UID: \"fb774bd5-e22c-4742-9d65-794519951feb\") " pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp"
May 20 15:45:04.917191 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:04.917159    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb774bd5-e22c-4742-9d65-794519951feb-proxy-tls\") pod \"model-chainer-6b9459f84c-nwvpp\" (UID: \"fb774bd5-e22c-4742-9d65-794519951feb\") " pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp"
May 20 15:45:05.213400 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:05.213312    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp"
May 20 15:45:05.357301 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:05.357272    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp"]
May 20 15:45:05.358396 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:45:05.358372    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb774bd5_e22c_4742_9d65_794519951feb.slice/crio-93ffe866ab397f500e7dc2417ec572c8bb91188b6afa76787b8250d3bd1fbdcf WatchSource:0}: Error finding container 93ffe866ab397f500e7dc2417ec572c8bb91188b6afa76787b8250d3bd1fbdcf: Status 404 returned error can't find the container with id 93ffe866ab397f500e7dc2417ec572c8bb91188b6afa76787b8250d3bd1fbdcf
May 20 15:45:05.424283 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:05.424248    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp" event={"ID":"fb774bd5-e22c-4742-9d65-794519951feb","Type":"ContainerStarted","Data":"93ffe866ab397f500e7dc2417ec572c8bb91188b6afa76787b8250d3bd1fbdcf"}
May 20 15:45:06.330453 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:06.330411    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg" podUID="1a538143-ec1a-49c3-9204-e35cf7adbd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 20 15:45:06.428824 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:06.428786    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp" event={"ID":"fb774bd5-e22c-4742-9d65-794519951feb","Type":"ContainerStarted","Data":"f5a4881fda2fe424741172217fbc9c0b95209e70630f52d1be2a755f73627aa5"}
May 20 15:45:06.429284 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:06.428865    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp"
May 20 15:45:06.457151 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:06.457052    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp" podStartSLOduration=2.457011558 podStartE2EDuration="2.457011558s" podCreationTimestamp="2026-05-20 15:45:04 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:45:06.456931493 +0000 UTC m=+815.395773050" watchObservedRunningTime="2026-05-20 15:45:06.457011558 +0000 UTC m=+815.395853114"
May 20 15:45:08.342666 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:08.342627    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5" podUID="4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 20 15:45:08.657819 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:08.657790    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk"
May 20 15:45:08.744902 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:08.744871    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b90118ab-d260-43ff-8b74-c492f22c2c6c-proxy-tls\") pod \"b90118ab-d260-43ff-8b74-c492f22c2c6c\" (UID: \"b90118ab-d260-43ff-8b74-c492f22c2c6c\") "
May 20 15:45:08.745095 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:08.744919    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b90118ab-d260-43ff-8b74-c492f22c2c6c-openshift-service-ca-bundle\") pod \"b90118ab-d260-43ff-8b74-c492f22c2c6c\" (UID: \"b90118ab-d260-43ff-8b74-c492f22c2c6c\") "
May 20 15:45:08.745338 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:08.745302    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b90118ab-d260-43ff-8b74-c492f22c2c6c-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "b90118ab-d260-43ff-8b74-c492f22c2c6c" (UID: "b90118ab-d260-43ff-8b74-c492f22c2c6c"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:45:08.747044 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:08.747004    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b90118ab-d260-43ff-8b74-c492f22c2c6c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b90118ab-d260-43ff-8b74-c492f22c2c6c" (UID: "b90118ab-d260-43ff-8b74-c492f22c2c6c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:45:08.845538 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:08.845502    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b90118ab-d260-43ff-8b74-c492f22c2c6c-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:45:08.845538 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:08.845533    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b90118ab-d260-43ff-8b74-c492f22c2c6c-openshift-service-ca-bundle\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:45:09.440094 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:09.440054    2568 generic.go:358] "Generic (PLEG): container finished" podID="b90118ab-d260-43ff-8b74-c492f22c2c6c" containerID="be53ca626d6abdb9a1fff0af8b54976e580058d3b53b69dc32c56177edcb8e95" exitCode=137
May 20 15:45:09.440481 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:09.440156    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk" event={"ID":"b90118ab-d260-43ff-8b74-c492f22c2c6c","Type":"ContainerDied","Data":"be53ca626d6abdb9a1fff0af8b54976e580058d3b53b69dc32c56177edcb8e95"}
May 20 15:45:09.440481 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:09.440169    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk"
May 20 15:45:09.440481 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:09.440194    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk" event={"ID":"b90118ab-d260-43ff-8b74-c492f22c2c6c","Type":"ContainerDied","Data":"77588b3bd204e8a81662cbaea86272d76002c676f1f5d2cbdcb8b20c8c686224"}
May 20 15:45:09.440481 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:09.440217    2568 scope.go:117] "RemoveContainer" containerID="be53ca626d6abdb9a1fff0af8b54976e580058d3b53b69dc32c56177edcb8e95"
May 20 15:45:09.448972 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:09.448954    2568 scope.go:117] "RemoveContainer" containerID="be53ca626d6abdb9a1fff0af8b54976e580058d3b53b69dc32c56177edcb8e95"
May 20 15:45:09.449258 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:45:09.449235    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be53ca626d6abdb9a1fff0af8b54976e580058d3b53b69dc32c56177edcb8e95\": container with ID starting with be53ca626d6abdb9a1fff0af8b54976e580058d3b53b69dc32c56177edcb8e95 not found: ID does not exist" containerID="be53ca626d6abdb9a1fff0af8b54976e580058d3b53b69dc32c56177edcb8e95"
May 20 15:45:09.449328 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:09.449270    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be53ca626d6abdb9a1fff0af8b54976e580058d3b53b69dc32c56177edcb8e95"} err="failed to get container status \"be53ca626d6abdb9a1fff0af8b54976e580058d3b53b69dc32c56177edcb8e95\": rpc error: code = NotFound desc = could not find container \"be53ca626d6abdb9a1fff0af8b54976e580058d3b53b69dc32c56177edcb8e95\": container with ID starting with be53ca626d6abdb9a1fff0af8b54976e580058d3b53b69dc32c56177edcb8e95 not found: ID does not exist"
May 20 15:45:09.466126 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:09.466098    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk"]
May 20 15:45:09.474346 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:09.474314    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-542af-d57596c4d-5w6hk"]
May 20 15:45:09.644397 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:09.644367    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b90118ab-d260-43ff-8b74-c492f22c2c6c" path="/var/lib/kubelet/pods/b90118ab-d260-43ff-8b74-c492f22c2c6c/volumes"
May 20 15:45:12.438358 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:12.438330    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp"
May 20 15:45:14.352797 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.352758    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp"]
May 20 15:45:14.353279 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.352990    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp" podUID="fb774bd5-e22c-4742-9d65-794519951feb" containerName="model-chainer" containerID="cri-o://f5a4881fda2fe424741172217fbc9c0b95209e70630f52d1be2a755f73627aa5" gracePeriod=30
May 20 15:45:14.568043 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.567984    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"]
May 20 15:45:14.568617 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.568597    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b90118ab-d260-43ff-8b74-c492f22c2c6c" containerName="switch-graph-542af"
May 20 15:45:14.568617 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.568619    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b90118ab-d260-43ff-8b74-c492f22c2c6c" containerName="switch-graph-542af"
May 20 15:45:14.568812 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.568736    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="b90118ab-d260-43ff-8b74-c492f22c2c6c" containerName="switch-graph-542af"
May 20 15:45:14.572044 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.571922    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"
May 20 15:45:14.575754 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.575238    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-23504-predictor-serving-cert\""
May 20 15:45:14.575754 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.575465    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-23504-kube-rbac-proxy-sar-config\""
May 20 15:45:14.589328 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.589301    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"]
May 20 15:45:14.596397 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.596373    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-23504-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7922d8ef-19c4-4ef9-a09b-f16db6cf33e5-success-200-isvc-23504-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-23504-predictor-557c78fdfd-f6js7\" (UID: \"7922d8ef-19c4-4ef9-a09b-f16db6cf33e5\") " pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"
May 20 15:45:14.596494 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.596415    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mffcb\" (UniqueName: \"kubernetes.io/projected/7922d8ef-19c4-4ef9-a09b-f16db6cf33e5-kube-api-access-mffcb\") pod \"success-200-isvc-23504-predictor-557c78fdfd-f6js7\" (UID: \"7922d8ef-19c4-4ef9-a09b-f16db6cf33e5\") " pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"
May 20 15:45:14.596494 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.596478    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7922d8ef-19c4-4ef9-a09b-f16db6cf33e5-proxy-tls\") pod \"success-200-isvc-23504-predictor-557c78fdfd-f6js7\" (UID: \"7922d8ef-19c4-4ef9-a09b-f16db6cf33e5\") " pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"
May 20 15:45:14.697468 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.697378    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-23504-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7922d8ef-19c4-4ef9-a09b-f16db6cf33e5-success-200-isvc-23504-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-23504-predictor-557c78fdfd-f6js7\" (UID: \"7922d8ef-19c4-4ef9-a09b-f16db6cf33e5\") " pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"
May 20 15:45:14.697468 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.697437    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mffcb\" (UniqueName: \"kubernetes.io/projected/7922d8ef-19c4-4ef9-a09b-f16db6cf33e5-kube-api-access-mffcb\") pod \"success-200-isvc-23504-predictor-557c78fdfd-f6js7\" (UID: \"7922d8ef-19c4-4ef9-a09b-f16db6cf33e5\") " pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"
May 20 15:45:14.697688 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.697629    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"]
May 20 15:45:14.697752 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.697700    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7922d8ef-19c4-4ef9-a09b-f16db6cf33e5-proxy-tls\") pod \"success-200-isvc-23504-predictor-557c78fdfd-f6js7\" (UID: \"7922d8ef-19c4-4ef9-a09b-f16db6cf33e5\") " pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"
May 20 15:45:14.697850 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:45:14.697834    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-23504-predictor-serving-cert: secret "success-200-isvc-23504-predictor-serving-cert" not found
May 20 15:45:14.697916 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:45:14.697904    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7922d8ef-19c4-4ef9-a09b-f16db6cf33e5-proxy-tls podName:7922d8ef-19c4-4ef9-a09b-f16db6cf33e5 nodeName:}" failed. No retries permitted until 2026-05-20 15:45:15.197883782 +0000 UTC m=+824.136725318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7922d8ef-19c4-4ef9-a09b-f16db6cf33e5-proxy-tls") pod "success-200-isvc-23504-predictor-557c78fdfd-f6js7" (UID: "7922d8ef-19c4-4ef9-a09b-f16db6cf33e5") : secret "success-200-isvc-23504-predictor-serving-cert" not found
May 20 15:45:14.698043 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.698000    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4" podUID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerName="kserve-container" containerID="cri-o://81c2b60d17c0f3a8c089f83410bb30251ff31756e0d3559fdcb79bde59a8269f" gracePeriod=30
May 20 15:45:14.698150 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.698134    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4" podUID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerName="kube-rbac-proxy" containerID="cri-o://623f2ddcf89e2da0fe028c6fdcae57b90d018f693350d4dda92b352852c7b575" gracePeriod=30
May 20 15:45:14.698215 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.698197    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-23504-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7922d8ef-19c4-4ef9-a09b-f16db6cf33e5-success-200-isvc-23504-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-23504-predictor-557c78fdfd-f6js7\" (UID: \"7922d8ef-19c4-4ef9-a09b-f16db6cf33e5\") " pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"
May 20 15:45:14.709740 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.709709    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"]
May 20 15:45:14.710538 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.710512    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mffcb\" (UniqueName: \"kubernetes.io/projected/7922d8ef-19c4-4ef9-a09b-f16db6cf33e5-kube-api-access-mffcb\") pod \"success-200-isvc-23504-predictor-557c78fdfd-f6js7\" (UID: \"7922d8ef-19c4-4ef9-a09b-f16db6cf33e5\") " pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"
May 20 15:45:14.713091 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.713062    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"
May 20 15:45:14.715791 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.715770    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-23504-predictor-serving-cert\""
May 20 15:45:14.717764 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.717747    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-23504-kube-rbac-proxy-sar-config\""
May 20 15:45:14.725147 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.725122    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"]
May 20 15:45:14.798867 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.798832    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-23504-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45e8ec2e-cb37-46fd-9bca-db9bcd707e33-error-404-isvc-23504-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-23504-predictor-6d476745d5-rhhv6\" (UID: \"45e8ec2e-cb37-46fd-9bca-db9bcd707e33\") " pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"
May 20 15:45:14.799088 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.798900    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77rd7\" (UniqueName: \"kubernetes.io/projected/45e8ec2e-cb37-46fd-9bca-db9bcd707e33-kube-api-access-77rd7\") pod \"error-404-isvc-23504-predictor-6d476745d5-rhhv6\" (UID: \"45e8ec2e-cb37-46fd-9bca-db9bcd707e33\") " pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"
May 20 15:45:14.799088 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.799001    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45e8ec2e-cb37-46fd-9bca-db9bcd707e33-proxy-tls\") pod \"error-404-isvc-23504-predictor-6d476745d5-rhhv6\" (UID: \"45e8ec2e-cb37-46fd-9bca-db9bcd707e33\") " pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"
May 20 15:45:14.899564 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.899527    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45e8ec2e-cb37-46fd-9bca-db9bcd707e33-proxy-tls\") pod \"error-404-isvc-23504-predictor-6d476745d5-rhhv6\" (UID: \"45e8ec2e-cb37-46fd-9bca-db9bcd707e33\") " pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"
May 20 15:45:14.899766 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.899604    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-23504-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45e8ec2e-cb37-46fd-9bca-db9bcd707e33-error-404-isvc-23504-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-23504-predictor-6d476745d5-rhhv6\" (UID: \"45e8ec2e-cb37-46fd-9bca-db9bcd707e33\") " pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"
May 20 15:45:14.899766 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.899667    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-77rd7\" (UniqueName: \"kubernetes.io/projected/45e8ec2e-cb37-46fd-9bca-db9bcd707e33-kube-api-access-77rd7\") pod \"error-404-isvc-23504-predictor-6d476745d5-rhhv6\" (UID: \"45e8ec2e-cb37-46fd-9bca-db9bcd707e33\") " pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"
May 20 15:45:14.900500 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.900469    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-23504-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45e8ec2e-cb37-46fd-9bca-db9bcd707e33-error-404-isvc-23504-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-23504-predictor-6d476745d5-rhhv6\" (UID: \"45e8ec2e-cb37-46fd-9bca-db9bcd707e33\") " pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"
May 20 15:45:14.902240 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.902215    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45e8ec2e-cb37-46fd-9bca-db9bcd707e33-proxy-tls\") pod \"error-404-isvc-23504-predictor-6d476745d5-rhhv6\" (UID: \"45e8ec2e-cb37-46fd-9bca-db9bcd707e33\") " pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"
May 20 15:45:14.909588 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:14.909563    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77rd7\" (UniqueName: \"kubernetes.io/projected/45e8ec2e-cb37-46fd-9bca-db9bcd707e33-kube-api-access-77rd7\") pod \"error-404-isvc-23504-predictor-6d476745d5-rhhv6\" (UID: \"45e8ec2e-cb37-46fd-9bca-db9bcd707e33\") " pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"
May 20 15:45:15.034335 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:15.034245    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"
May 20 15:45:15.176300 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:15.176270    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"]
May 20 15:45:15.178731 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:45:15.178704    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e8ec2e_cb37_46fd_9bca_db9bcd707e33.slice/crio-70181150ff39f2d998c34fd4a94c1d600e5152c716b666e5074494d81f9cf1ab WatchSource:0}: Error finding container 70181150ff39f2d998c34fd4a94c1d600e5152c716b666e5074494d81f9cf1ab: Status 404 returned error can't find the container with id 70181150ff39f2d998c34fd4a94c1d600e5152c716b666e5074494d81f9cf1ab
May 20 15:45:15.202537 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:15.202510    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7922d8ef-19c4-4ef9-a09b-f16db6cf33e5-proxy-tls\") pod \"success-200-isvc-23504-predictor-557c78fdfd-f6js7\" (UID: \"7922d8ef-19c4-4ef9-a09b-f16db6cf33e5\") " pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"
May 20 15:45:15.204923 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:15.204905    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7922d8ef-19c4-4ef9-a09b-f16db6cf33e5-proxy-tls\") pod \"success-200-isvc-23504-predictor-557c78fdfd-f6js7\" (UID: \"7922d8ef-19c4-4ef9-a09b-f16db6cf33e5\") " pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"
May 20 15:45:15.462553 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:15.462518    2568 generic.go:358] "Generic (PLEG): container finished" podID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerID="623f2ddcf89e2da0fe028c6fdcae57b90d018f693350d4dda92b352852c7b575" exitCode=2
May 20 15:45:15.463014 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:15.462598    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4" event={"ID":"15b10f19-0c65-4064-b0d7-7213aa008b0b","Type":"ContainerDied","Data":"623f2ddcf89e2da0fe028c6fdcae57b90d018f693350d4dda92b352852c7b575"}
May 20 15:45:15.464142 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:15.464119    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6" event={"ID":"45e8ec2e-cb37-46fd-9bca-db9bcd707e33","Type":"ContainerStarted","Data":"ac3a54b558db5fcef62aab9c24890ce2233032a8ba2eca8f7aa73297fe90924c"}
May 20 15:45:15.464252 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:15.464144    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6" event={"ID":"45e8ec2e-cb37-46fd-9bca-db9bcd707e33","Type":"ContainerStarted","Data":"74c2486c69dab3b26ad1e6fbb2b88bfbeb1abdb9660e118e1fce209ec41451de"}
May 20 15:45:15.464252 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:15.464157    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6" event={"ID":"45e8ec2e-cb37-46fd-9bca-db9bcd707e33","Type":"ContainerStarted","Data":"70181150ff39f2d998c34fd4a94c1d600e5152c716b666e5074494d81f9cf1ab"}
May 20 15:45:15.464252 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:15.464194    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"
May 20 15:45:15.484554 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:15.484507    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6" podStartSLOduration=1.484492827 podStartE2EDuration="1.484492827s" podCreationTimestamp="2026-05-20 15:45:14 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:45:15.483943166 +0000 UTC m=+824.422784723" watchObservedRunningTime="2026-05-20 15:45:15.484492827 +0000 UTC m=+824.423334454"
May 20 15:45:15.486055 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:15.486035    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"
May 20 15:45:15.634322 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:15.634299    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"]
May 20 15:45:15.636342 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:45:15.636317    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7922d8ef_19c4_4ef9_a09b_f16db6cf33e5.slice/crio-8c323eac01dd3d7ea2e3313f2b10e69f304fb8a23a1f485804a4df02d7219eb4 WatchSource:0}: Error finding container 8c323eac01dd3d7ea2e3313f2b10e69f304fb8a23a1f485804a4df02d7219eb4: Status 404 returned error can't find the container with id 8c323eac01dd3d7ea2e3313f2b10e69f304fb8a23a1f485804a4df02d7219eb4
May 20 15:45:16.331050 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:16.330996    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg" podUID="1a538143-ec1a-49c3-9204-e35cf7adbd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 20 15:45:16.469664 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:16.469578    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7" event={"ID":"7922d8ef-19c4-4ef9-a09b-f16db6cf33e5","Type":"ContainerStarted","Data":"37ff1e4e649c408617b071e004fb1e8c7799cf02d8fba1857d75ae7cf9520ae0"}
May 20 15:45:16.469664 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:16.469628    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7" event={"ID":"7922d8ef-19c4-4ef9-a09b-f16db6cf33e5","Type":"ContainerStarted","Data":"19dcbe5fb0f248c40b43620817e47f97fb47fd189e087afa1e8b252727df2f85"}
May 20 15:45:16.469664 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:16.469643    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7" event={"ID":"7922d8ef-19c4-4ef9-a09b-f16db6cf33e5","Type":"ContainerStarted","Data":"8c323eac01dd3d7ea2e3313f2b10e69f304fb8a23a1f485804a4df02d7219eb4"}
May 20 15:45:16.470279 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:16.470119    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"
May 20 15:45:16.470279 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:16.470147    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"
May 20 15:45:16.470640 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:16.470600    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"
May 20 15:45:16.471534 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:16.471495    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7" podUID="7922d8ef-19c4-4ef9-a09b-f16db6cf33e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 20 15:45:16.471766 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:16.471738    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6" podUID="45e8ec2e-cb37-46fd-9bca-db9bcd707e33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
May 20 15:45:16.493222 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:16.493169    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7" podStartSLOduration=2.49315438 podStartE2EDuration="2.49315438s" podCreationTimestamp="2026-05-20 15:45:14 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:45:16.489957409 +0000 UTC m=+825.428798966" watchObservedRunningTime="2026-05-20 15:45:16.49315438 +0000 UTC m=+825.431995934"
May 20 15:45:17.436854 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:17.436815    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp" podUID="fb774bd5-e22c-4742-9d65-794519951feb" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:45:17.474489 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:17.474444    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6" podUID="45e8ec2e-cb37-46fd-9bca-db9bcd707e33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
May 20 15:45:17.474947 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:17.474624    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7" podUID="7922d8ef-19c4-4ef9-a09b-f16db6cf33e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 20 15:45:18.105964 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:18.105914    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4" podUID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
May 20 15:45:18.111891 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:18.111864    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4" podUID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 20 15:45:18.342810 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:18.342774    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5" podUID="4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 20 15:45:18.479466 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:18.479420    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7" podUID="7922d8ef-19c4-4ef9-a09b-f16db6cf33e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 20 15:45:18.645214 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:18.645190    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"
May 20 15:45:18.739702 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:18.739616    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7tnp\" (UniqueName: \"kubernetes.io/projected/15b10f19-0c65-4064-b0d7-7213aa008b0b-kube-api-access-w7tnp\") pod \"15b10f19-0c65-4064-b0d7-7213aa008b0b\" (UID: \"15b10f19-0c65-4064-b0d7-7213aa008b0b\") "
May 20 15:45:18.739702 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:18.739660    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15b10f19-0c65-4064-b0d7-7213aa008b0b-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"15b10f19-0c65-4064-b0d7-7213aa008b0b\" (UID: \"15b10f19-0c65-4064-b0d7-7213aa008b0b\") "
May 20 15:45:18.739702 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:18.739698    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15b10f19-0c65-4064-b0d7-7213aa008b0b-kserve-provision-location\") pod \"15b10f19-0c65-4064-b0d7-7213aa008b0b\" (UID: \"15b10f19-0c65-4064-b0d7-7213aa008b0b\") "
May 20 15:45:18.739984 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:18.739752    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15b10f19-0c65-4064-b0d7-7213aa008b0b-proxy-tls\") pod \"15b10f19-0c65-4064-b0d7-7213aa008b0b\" (UID: \"15b10f19-0c65-4064-b0d7-7213aa008b0b\") "
May 20 15:45:18.740131 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:18.740101    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b10f19-0c65-4064-b0d7-7213aa008b0b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "15b10f19-0c65-4064-b0d7-7213aa008b0b" (UID: "15b10f19-0c65-4064-b0d7-7213aa008b0b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:45:18.740185 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:18.740129    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b10f19-0c65-4064-b0d7-7213aa008b0b-isvc-xgboost-graph-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-graph-kube-rbac-proxy-sar-config") pod "15b10f19-0c65-4064-b0d7-7213aa008b0b" (UID: "15b10f19-0c65-4064-b0d7-7213aa008b0b"). InnerVolumeSpecName "isvc-xgboost-graph-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:45:18.741884 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:18.741860    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b10f19-0c65-4064-b0d7-7213aa008b0b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "15b10f19-0c65-4064-b0d7-7213aa008b0b" (UID: "15b10f19-0c65-4064-b0d7-7213aa008b0b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:45:18.741884 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:18.741869    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b10f19-0c65-4064-b0d7-7213aa008b0b-kube-api-access-w7tnp" (OuterVolumeSpecName: "kube-api-access-w7tnp") pod "15b10f19-0c65-4064-b0d7-7213aa008b0b" (UID: "15b10f19-0c65-4064-b0d7-7213aa008b0b"). InnerVolumeSpecName "kube-api-access-w7tnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:45:18.840626 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:18.840581    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15b10f19-0c65-4064-b0d7-7213aa008b0b-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:45:18.840626 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:18.840621    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7tnp\" (UniqueName: \"kubernetes.io/projected/15b10f19-0c65-4064-b0d7-7213aa008b0b-kube-api-access-w7tnp\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:45:18.840826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:18.840638    2568 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15b10f19-0c65-4064-b0d7-7213aa008b0b-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:45:18.840826 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:18.840652    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15b10f19-0c65-4064-b0d7-7213aa008b0b-kserve-provision-location\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:45:19.485980 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:19.485944    2568 generic.go:358] "Generic (PLEG): container finished" podID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerID="81c2b60d17c0f3a8c089f83410bb30251ff31756e0d3559fdcb79bde59a8269f" exitCode=0
May 20 15:45:19.486414 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:19.485996    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4" event={"ID":"15b10f19-0c65-4064-b0d7-7213aa008b0b","Type":"ContainerDied","Data":"81c2b60d17c0f3a8c089f83410bb30251ff31756e0d3559fdcb79bde59a8269f"}
May 20 15:45:19.486414 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:19.486049    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"
May 20 15:45:19.486414 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:19.486066    2568 scope.go:117] "RemoveContainer" containerID="623f2ddcf89e2da0fe028c6fdcae57b90d018f693350d4dda92b352852c7b575"
May 20 15:45:19.486414 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:19.486054    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4" event={"ID":"15b10f19-0c65-4064-b0d7-7213aa008b0b","Type":"ContainerDied","Data":"88ec3bf872796be33a959ca7a45f601093007af070ed6f537bb2df0915e60db1"}
May 20 15:45:19.495131 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:19.495107    2568 scope.go:117] "RemoveContainer" containerID="81c2b60d17c0f3a8c089f83410bb30251ff31756e0d3559fdcb79bde59a8269f"
May 20 15:45:19.505270 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:19.505249    2568 scope.go:117] "RemoveContainer" containerID="fa964618baec4f9db6d5e6ab764d99fe5c555648c62bca5564fe7334fb13c4b4"
May 20 15:45:19.514447 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:19.514413    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"]
May 20 15:45:19.515608 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:19.514933    2568 scope.go:117] "RemoveContainer" containerID="623f2ddcf89e2da0fe028c6fdcae57b90d018f693350d4dda92b352852c7b575"
May 20 15:45:19.515608 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:45:19.515474    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"623f2ddcf89e2da0fe028c6fdcae57b90d018f693350d4dda92b352852c7b575\": container with ID starting with 623f2ddcf89e2da0fe028c6fdcae57b90d018f693350d4dda92b352852c7b575 not found: ID does not exist" containerID="623f2ddcf89e2da0fe028c6fdcae57b90d018f693350d4dda92b352852c7b575"
May 20 15:45:19.515608 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:19.515529    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623f2ddcf89e2da0fe028c6fdcae57b90d018f693350d4dda92b352852c7b575"} err="failed to get container status \"623f2ddcf89e2da0fe028c6fdcae57b90d018f693350d4dda92b352852c7b575\": rpc error: code = NotFound desc = could not find container \"623f2ddcf89e2da0fe028c6fdcae57b90d018f693350d4dda92b352852c7b575\": container with ID starting with 623f2ddcf89e2da0fe028c6fdcae57b90d018f693350d4dda92b352852c7b575 not found: ID does not exist"
May 20 15:45:19.515608 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:19.515558    2568 scope.go:117] "RemoveContainer" containerID="81c2b60d17c0f3a8c089f83410bb30251ff31756e0d3559fdcb79bde59a8269f"
May 20 15:45:19.515843 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:45:19.515823    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c2b60d17c0f3a8c089f83410bb30251ff31756e0d3559fdcb79bde59a8269f\": container with ID starting with 81c2b60d17c0f3a8c089f83410bb30251ff31756e0d3559fdcb79bde59a8269f not found: ID does not exist" containerID="81c2b60d17c0f3a8c089f83410bb30251ff31756e0d3559fdcb79bde59a8269f"
May 20 15:45:19.515887 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:19.515850    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c2b60d17c0f3a8c089f83410bb30251ff31756e0d3559fdcb79bde59a8269f"} err="failed to get container status \"81c2b60d17c0f3a8c089f83410bb30251ff31756e0d3559fdcb79bde59a8269f\": rpc error: code = NotFound desc = could not find container \"81c2b60d17c0f3a8c089f83410bb30251ff31756e0d3559fdcb79bde59a8269f\": container with ID starting with 81c2b60d17c0f3a8c089f83410bb30251ff31756e0d3559fdcb79bde59a8269f not found: ID does not exist"
May 20 15:45:19.515887 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:19.515865    2568 scope.go:117] "RemoveContainer" containerID="fa964618baec4f9db6d5e6ab764d99fe5c555648c62bca5564fe7334fb13c4b4"
May 20 15:45:19.516178 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:45:19.516160    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa964618baec4f9db6d5e6ab764d99fe5c555648c62bca5564fe7334fb13c4b4\": container with ID starting with fa964618baec4f9db6d5e6ab764d99fe5c555648c62bca5564fe7334fb13c4b4 not found: ID does not exist" containerID="fa964618baec4f9db6d5e6ab764d99fe5c555648c62bca5564fe7334fb13c4b4"
May 20 15:45:19.516248 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:19.516184    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa964618baec4f9db6d5e6ab764d99fe5c555648c62bca5564fe7334fb13c4b4"} err="failed to get container status \"fa964618baec4f9db6d5e6ab764d99fe5c555648c62bca5564fe7334fb13c4b4\": rpc error: code = NotFound desc = could not find container \"fa964618baec4f9db6d5e6ab764d99fe5c555648c62bca5564fe7334fb13c4b4\": container with ID starting with fa964618baec4f9db6d5e6ab764d99fe5c555648c62bca5564fe7334fb13c4b4 not found: ID does not exist"
May 20 15:45:19.517553 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:19.517525    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-h8dq4"]
May 20 15:45:19.644945 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:19.644911    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15b10f19-0c65-4064-b0d7-7213aa008b0b" path="/var/lib/kubelet/pods/15b10f19-0c65-4064-b0d7-7213aa008b0b/volumes"
May 20 15:45:22.437264 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:22.437222    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp" podUID="fb774bd5-e22c-4742-9d65-794519951feb" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:45:22.480183 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:22.480154    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"
May 20 15:45:22.480752 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:22.480720    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6" podUID="45e8ec2e-cb37-46fd-9bca-db9bcd707e33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
May 20 15:45:23.484031 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:23.483933    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"
May 20 15:45:23.484580 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:23.484551    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7" podUID="7922d8ef-19c4-4ef9-a09b-f16db6cf33e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 20 15:45:26.330457 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:26.330421    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg" podUID="1a538143-ec1a-49c3-9204-e35cf7adbd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 20 15:45:27.437151 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:27.437097    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp" podUID="fb774bd5-e22c-4742-9d65-794519951feb" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:45:27.437539 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:27.437217    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp"
May 20 15:45:28.342956 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:28.342920    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5" podUID="4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 20 15:45:32.436433 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:32.436393    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp" podUID="fb774bd5-e22c-4742-9d65-794519951feb" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:45:32.481572 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:32.481534    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6" podUID="45e8ec2e-cb37-46fd-9bca-db9bcd707e33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
May 20 15:45:33.484514 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:33.484477    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7" podUID="7922d8ef-19c4-4ef9-a09b-f16db6cf33e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 20 15:45:36.331221 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:36.331190    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"
May 20 15:45:37.436853 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:37.436815    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp" podUID="fb774bd5-e22c-4742-9d65-794519951feb" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:45:38.342774 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:38.342745    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"
May 20 15:45:42.436588 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:42.436552    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp" podUID="fb774bd5-e22c-4742-9d65-794519951feb" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:45:42.481248 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:42.481206    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6" podUID="45e8ec2e-cb37-46fd-9bca-db9bcd707e33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
May 20 15:45:43.484812 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:43.484770    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7" podUID="7922d8ef-19c4-4ef9-a09b-f16db6cf33e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 20 15:45:44.388591 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:45:44.388554    2568 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb774bd5_e22c_4742_9d65_794519951feb.slice/crio-conmon-f5a4881fda2fe424741172217fbc9c0b95209e70630f52d1be2a755f73627aa5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb774bd5_e22c_4742_9d65_794519951feb.slice/crio-f5a4881fda2fe424741172217fbc9c0b95209e70630f52d1be2a755f73627aa5.scope\": RecentStats: unable to find data in memory cache]"
May 20 15:45:44.388753 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:45:44.388623    2568 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb774bd5_e22c_4742_9d65_794519951feb.slice/crio-conmon-f5a4881fda2fe424741172217fbc9c0b95209e70630f52d1be2a755f73627aa5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb774bd5_e22c_4742_9d65_794519951feb.slice/crio-f5a4881fda2fe424741172217fbc9c0b95209e70630f52d1be2a755f73627aa5.scope\": RecentStats: unable to find data in memory cache]"
May 20 15:45:44.388753 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:45:44.388719    2568 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb774bd5_e22c_4742_9d65_794519951feb.slice/crio-conmon-f5a4881fda2fe424741172217fbc9c0b95209e70630f52d1be2a755f73627aa5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb774bd5_e22c_4742_9d65_794519951feb.slice/crio-f5a4881fda2fe424741172217fbc9c0b95209e70630f52d1be2a755f73627aa5.scope\": RecentStats: unable to find data in memory cache]"
May 20 15:45:44.388873 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:45:44.388846    2568 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb774bd5_e22c_4742_9d65_794519951feb.slice/crio-conmon-f5a4881fda2fe424741172217fbc9c0b95209e70630f52d1be2a755f73627aa5.scope\": RecentStats: unable to find data in memory cache]"
May 20 15:45:44.520143 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:44.520114    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp"
May 20 15:45:44.578504 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:44.578471    2568 generic.go:358] "Generic (PLEG): container finished" podID="fb774bd5-e22c-4742-9d65-794519951feb" containerID="f5a4881fda2fe424741172217fbc9c0b95209e70630f52d1be2a755f73627aa5" exitCode=0
May 20 15:45:44.578639 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:44.578542    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp" event={"ID":"fb774bd5-e22c-4742-9d65-794519951feb","Type":"ContainerDied","Data":"f5a4881fda2fe424741172217fbc9c0b95209e70630f52d1be2a755f73627aa5"}
May 20 15:45:44.578639 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:44.578567    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp" event={"ID":"fb774bd5-e22c-4742-9d65-794519951feb","Type":"ContainerDied","Data":"93ffe866ab397f500e7dc2417ec572c8bb91188b6afa76787b8250d3bd1fbdcf"}
May 20 15:45:44.578639 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:44.578582    2568 scope.go:117] "RemoveContainer" containerID="f5a4881fda2fe424741172217fbc9c0b95209e70630f52d1be2a755f73627aa5"
May 20 15:45:44.578639 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:44.578546    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp"
May 20 15:45:44.586770 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:44.586754    2568 scope.go:117] "RemoveContainer" containerID="f5a4881fda2fe424741172217fbc9c0b95209e70630f52d1be2a755f73627aa5"
May 20 15:45:44.587062 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:45:44.587043    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a4881fda2fe424741172217fbc9c0b95209e70630f52d1be2a755f73627aa5\": container with ID starting with f5a4881fda2fe424741172217fbc9c0b95209e70630f52d1be2a755f73627aa5 not found: ID does not exist" containerID="f5a4881fda2fe424741172217fbc9c0b95209e70630f52d1be2a755f73627aa5"
May 20 15:45:44.587121 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:44.587070    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a4881fda2fe424741172217fbc9c0b95209e70630f52d1be2a755f73627aa5"} err="failed to get container status \"f5a4881fda2fe424741172217fbc9c0b95209e70630f52d1be2a755f73627aa5\": rpc error: code = NotFound desc = could not find container \"f5a4881fda2fe424741172217fbc9c0b95209e70630f52d1be2a755f73627aa5\": container with ID starting with f5a4881fda2fe424741172217fbc9c0b95209e70630f52d1be2a755f73627aa5 not found: ID does not exist"
May 20 15:45:44.675927 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:44.675850    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb774bd5-e22c-4742-9d65-794519951feb-openshift-service-ca-bundle\") pod \"fb774bd5-e22c-4742-9d65-794519951feb\" (UID: \"fb774bd5-e22c-4742-9d65-794519951feb\") "
May 20 15:45:44.675927 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:44.675904    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb774bd5-e22c-4742-9d65-794519951feb-proxy-tls\") pod \"fb774bd5-e22c-4742-9d65-794519951feb\" (UID: \"fb774bd5-e22c-4742-9d65-794519951feb\") "
May 20 15:45:44.676348 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:44.676321    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb774bd5-e22c-4742-9d65-794519951feb-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "fb774bd5-e22c-4742-9d65-794519951feb" (UID: "fb774bd5-e22c-4742-9d65-794519951feb"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:45:44.678206 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:44.678182    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb774bd5-e22c-4742-9d65-794519951feb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb774bd5-e22c-4742-9d65-794519951feb" (UID: "fb774bd5-e22c-4742-9d65-794519951feb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:45:44.776522 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:44.776484    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb774bd5-e22c-4742-9d65-794519951feb-openshift-service-ca-bundle\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:45:44.776522 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:44.776516    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb774bd5-e22c-4742-9d65-794519951feb-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:45:44.899582 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:44.899550    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp"]
May 20 15:45:44.904922 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:44.904894    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6b9459f84c-nwvpp"]
May 20 15:45:45.643784 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:45.643750    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb774bd5-e22c-4742-9d65-794519951feb" path="/var/lib/kubelet/pods/fb774bd5-e22c-4742-9d65-794519951feb/volumes"
May 20 15:45:48.808392 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:48.808354    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c"]
May 20 15:45:48.809050 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:48.809002    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb774bd5-e22c-4742-9d65-794519951feb" containerName="model-chainer"
May 20 15:45:48.809050 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:48.809043    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb774bd5-e22c-4742-9d65-794519951feb" containerName="model-chainer"
May 20 15:45:48.809225 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:48.809064    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerName="storage-initializer"
May 20 15:45:48.809225 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:48.809072    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerName="storage-initializer"
May 20 15:45:48.809225 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:48.809091    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerName="kube-rbac-proxy"
May 20 15:45:48.809225 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:48.809099    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerName="kube-rbac-proxy"
May 20 15:45:48.809225 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:48.809116    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerName="kserve-container"
May 20 15:45:48.809225 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:48.809123    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerName="kserve-container"
May 20 15:45:48.809225 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:48.809218    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="fb774bd5-e22c-4742-9d65-794519951feb" containerName="model-chainer"
May 20 15:45:48.809556 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:48.809230    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerName="kserve-container"
May 20 15:45:48.809556 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:48.809241    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="15b10f19-0c65-4064-b0d7-7213aa008b0b" containerName="kube-rbac-proxy"
May 20 15:45:48.814097 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:48.814076    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c"
May 20 15:45:48.816856 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:48.816836    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-d4d66-serving-cert\""
May 20 15:45:48.816954 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:48.816842    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-d4d66-kube-rbac-proxy-sar-config\""
May 20 15:45:48.827725 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:48.827701    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c"]
May 20 15:45:48.910938 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:48.910908    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ba7204-c40a-4fdc-97de-6257e21c8155-openshift-service-ca-bundle\") pod \"switch-graph-d4d66-58988c469-qj57c\" (UID: \"82ba7204-c40a-4fdc-97de-6257e21c8155\") " pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c"
May 20 15:45:48.911115 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:48.910953    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82ba7204-c40a-4fdc-97de-6257e21c8155-proxy-tls\") pod \"switch-graph-d4d66-58988c469-qj57c\" (UID: \"82ba7204-c40a-4fdc-97de-6257e21c8155\") " pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c"
May 20 15:45:49.011851 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:49.011824    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82ba7204-c40a-4fdc-97de-6257e21c8155-proxy-tls\") pod \"switch-graph-d4d66-58988c469-qj57c\" (UID: \"82ba7204-c40a-4fdc-97de-6257e21c8155\") " pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c"
May 20 15:45:49.011985 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:49.011915    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ba7204-c40a-4fdc-97de-6257e21c8155-openshift-service-ca-bundle\") pod \"switch-graph-d4d66-58988c469-qj57c\" (UID: \"82ba7204-c40a-4fdc-97de-6257e21c8155\") " pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c"
May 20 15:45:49.012500 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:49.012483    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ba7204-c40a-4fdc-97de-6257e21c8155-openshift-service-ca-bundle\") pod \"switch-graph-d4d66-58988c469-qj57c\" (UID: \"82ba7204-c40a-4fdc-97de-6257e21c8155\") " pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c"
May 20 15:45:49.014382 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:49.014358    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82ba7204-c40a-4fdc-97de-6257e21c8155-proxy-tls\") pod \"switch-graph-d4d66-58988c469-qj57c\" (UID: \"82ba7204-c40a-4fdc-97de-6257e21c8155\") " pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c"
May 20 15:45:49.124063 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:49.123975    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c"
May 20 15:45:49.266372 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:49.266341    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c"]
May 20 15:45:49.270463 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:45:49.270422    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ba7204_c40a_4fdc_97de_6257e21c8155.slice/crio-4b4605cca90cd72915080bc92ca82dfa2992286ef5a73af0f0a3336db6347b03 WatchSource:0}: Error finding container 4b4605cca90cd72915080bc92ca82dfa2992286ef5a73af0f0a3336db6347b03: Status 404 returned error can't find the container with id 4b4605cca90cd72915080bc92ca82dfa2992286ef5a73af0f0a3336db6347b03
May 20 15:45:49.600183 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:49.600145    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c" event={"ID":"82ba7204-c40a-4fdc-97de-6257e21c8155","Type":"ContainerStarted","Data":"6a9f9e19972e327a0634387ab9476f6bec7da6a0f853aa7f98b9f585b2621af4"}
May 20 15:45:49.600357 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:49.600190    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c" event={"ID":"82ba7204-c40a-4fdc-97de-6257e21c8155","Type":"ContainerStarted","Data":"4b4605cca90cd72915080bc92ca82dfa2992286ef5a73af0f0a3336db6347b03"}
May 20 15:45:49.600357 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:49.600268    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c"
May 20 15:45:49.618395 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:49.618355    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c" podStartSLOduration=1.618341872 podStartE2EDuration="1.618341872s" podCreationTimestamp="2026-05-20 15:45:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:45:49.616189884 +0000 UTC m=+858.555031441" watchObservedRunningTime="2026-05-20 15:45:49.618341872 +0000 UTC m=+858.557183428"
May 20 15:45:52.481250 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:52.481211    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6" podUID="45e8ec2e-cb37-46fd-9bca-db9bcd707e33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
May 20 15:45:53.484635 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:53.484596    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7" podUID="7922d8ef-19c4-4ef9-a09b-f16db6cf33e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 20 15:45:55.609597 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:45:55.609569    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c"
May 20 15:46:02.482046 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:46:02.481993    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"
May 20 15:46:03.485175 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:46:03.485150    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"
May 20 15:46:14.551169 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:46:14.551134    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29"]
May 20 15:46:14.556032 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:46:14.555984    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29"
May 20 15:46:14.560401 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:46:14.560302    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-23504-serving-cert\""
May 20 15:46:14.560401 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:46:14.560376    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-23504-kube-rbac-proxy-sar-config\""
May 20 15:46:14.564984 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:46:14.564963    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29"]
May 20 15:46:14.634370 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:46:14.634341    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d23dde2-c954-42ec-84b2-2686d2e578ad-proxy-tls\") pod \"sequence-graph-23504-54f4b4d8bc-pqd29\" (UID: \"7d23dde2-c954-42ec-84b2-2686d2e578ad\") " pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29"
May 20 15:46:14.634542 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:46:14.634389    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d23dde2-c954-42ec-84b2-2686d2e578ad-openshift-service-ca-bundle\") pod \"sequence-graph-23504-54f4b4d8bc-pqd29\" (UID: \"7d23dde2-c954-42ec-84b2-2686d2e578ad\") " pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29"
May 20 15:46:14.735594 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:46:14.735546    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d23dde2-c954-42ec-84b2-2686d2e578ad-openshift-service-ca-bundle\") pod \"sequence-graph-23504-54f4b4d8bc-pqd29\" (UID: \"7d23dde2-c954-42ec-84b2-2686d2e578ad\") " pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29"
May 20 15:46:14.735803 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:46:14.735670    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d23dde2-c954-42ec-84b2-2686d2e578ad-proxy-tls\") pod \"sequence-graph-23504-54f4b4d8bc-pqd29\" (UID: \"7d23dde2-c954-42ec-84b2-2686d2e578ad\") " pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29"
May 20 15:46:14.736260 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:46:14.736234    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d23dde2-c954-42ec-84b2-2686d2e578ad-openshift-service-ca-bundle\") pod \"sequence-graph-23504-54f4b4d8bc-pqd29\" (UID: \"7d23dde2-c954-42ec-84b2-2686d2e578ad\") " pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29"
May 20 15:46:14.738298 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:46:14.738276    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d23dde2-c954-42ec-84b2-2686d2e578ad-proxy-tls\") pod \"sequence-graph-23504-54f4b4d8bc-pqd29\" (UID: \"7d23dde2-c954-42ec-84b2-2686d2e578ad\") " pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29"
May 20 15:46:14.888800 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:46:14.888757    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29"
May 20 15:46:15.022590 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:46:15.022489    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29"]
May 20 15:46:15.690550 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:46:15.690508    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29" event={"ID":"7d23dde2-c954-42ec-84b2-2686d2e578ad","Type":"ContainerStarted","Data":"4045ca7d46594da939e0517d0d54e65b5485819637f0daa4004094e9e1d164ef"}
May 20 15:46:15.690550 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:46:15.690556    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29" event={"ID":"7d23dde2-c954-42ec-84b2-2686d2e578ad","Type":"ContainerStarted","Data":"bbcc6fc476dceb3280d89f4c2d6f20fa96237fc93fe8227699f3612b8303f03d"}
May 20 15:46:15.690999 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:46:15.690590    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29"
May 20 15:46:15.707465 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:46:15.707411    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29" podStartSLOduration=1.707391299 podStartE2EDuration="1.707391299s" podCreationTimestamp="2026-05-20 15:46:14 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:46:15.706401734 +0000 UTC m=+884.645243282" watchObservedRunningTime="2026-05-20 15:46:15.707391299 +0000 UTC m=+884.646232858"
May 20 15:46:21.701163 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:46:21.701136    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29"
May 20 15:46:31.617781 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:46:31.617755    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 15:46:31.617781 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:46:31.617767    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 15:49:31.656209 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:49:31.656180    2568 scope.go:117] "RemoveContainer" containerID="1a6b38bcb338616a7852cc7ff3348edaca955e7a6d2f8d9b1371ed37153c5642"
May 20 15:49:31.664961 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:49:31.664939    2568 scope.go:117] "RemoveContainer" containerID="c86f8f6e6f13ede72e5228d44ecedc28ece6489a514e1b9300a87308b621609d"
May 20 15:51:31.648011 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:51:31.647987    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 15:51:31.648603 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:51:31.648582    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 15:54:03.515132 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.515045    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c"]
May 20 15:54:03.515625 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.515339    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c" podUID="82ba7204-c40a-4fdc-97de-6257e21c8155" containerName="switch-graph-d4d66" containerID="cri-o://6a9f9e19972e327a0634387ab9476f6bec7da6a0f853aa7f98b9f585b2621af4" gracePeriod=30
May 20 15:54:03.622921 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.622891    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"]
May 20 15:54:03.623235 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.623201    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg" podUID="1a538143-ec1a-49c3-9204-e35cf7adbd2e" containerName="kserve-container" containerID="cri-o://efdfbb1c5bae91bb24aae5b5868374fd18cd8b110a0d828936f0e9a1c11ca75c" gracePeriod=30
May 20 15:54:03.623381 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.623271    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg" podUID="1a538143-ec1a-49c3-9204-e35cf7adbd2e" containerName="kube-rbac-proxy" containerID="cri-o://71831e30ba3629096a4bf0f9630318884c003e44988600665015f88b87e3cc86" gracePeriod=30
May 20 15:54:03.750976 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.750939    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"]
May 20 15:54:03.751250 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.751223    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5" podUID="4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b" containerName="kserve-container" containerID="cri-o://cf92198c9d9f5e9e63795e4e62194f097be5b09cec181c81a8521944b6b294c6" gracePeriod=30
May 20 15:54:03.751355 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.751303    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5" podUID="4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b" containerName="kube-rbac-proxy" containerID="cri-o://1bb6886bb8e627fbd06a2116f8278a7586f991d6b4e80df3bcd234e1738369ec" gracePeriod=30
May 20 15:54:03.769247 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.767764    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"]
May 20 15:54:03.782403 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.782373    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"
May 20 15:54:03.785319 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.785127    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-687f8-kube-rbac-proxy-sar-config\""
May 20 15:54:03.785457 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.785337    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-687f8-predictor-serving-cert\""
May 20 15:54:03.793304 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.792653    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"]
May 20 15:54:03.827993 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.827943    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"]
May 20 15:54:03.832502 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.832481    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"
May 20 15:54:03.836307 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.836279    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-687f8-predictor-serving-cert\""
May 20 15:54:03.836545 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.836523    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-687f8-kube-rbac-proxy-sar-config\""
May 20 15:54:03.838739 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.838714    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"]
May 20 15:54:03.876289 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.876254    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41bf512d-9315-457b-a451-d7fb1eb230d4-proxy-tls\") pod \"success-200-isvc-687f8-predictor-7d75c44d4d-64c45\" (UID: \"41bf512d-9315-457b-a451-d7fb1eb230d4\") " pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"
May 20 15:54:03.876443 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.876305    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-687f8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41bf512d-9315-457b-a451-d7fb1eb230d4-success-200-isvc-687f8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-687f8-predictor-7d75c44d4d-64c45\" (UID: \"41bf512d-9315-457b-a451-d7fb1eb230d4\") " pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"
May 20 15:54:03.876443 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.876377    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l96ht\" (UniqueName: \"kubernetes.io/projected/41bf512d-9315-457b-a451-d7fb1eb230d4-kube-api-access-l96ht\") pod \"success-200-isvc-687f8-predictor-7d75c44d4d-64c45\" (UID: \"41bf512d-9315-457b-a451-d7fb1eb230d4\") " pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"
May 20 15:54:03.977536 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.977496    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6vwk\" (UniqueName: \"kubernetes.io/projected/16b42b9f-fa74-4e57-8b69-f166541af5bd-kube-api-access-w6vwk\") pod \"error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw\" (UID: \"16b42b9f-fa74-4e57-8b69-f166541af5bd\") " pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"
May 20 15:54:03.977536 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.977539    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41bf512d-9315-457b-a451-d7fb1eb230d4-proxy-tls\") pod \"success-200-isvc-687f8-predictor-7d75c44d4d-64c45\" (UID: \"41bf512d-9315-457b-a451-d7fb1eb230d4\") " pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"
May 20 15:54:03.977844 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.977567    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-687f8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16b42b9f-fa74-4e57-8b69-f166541af5bd-error-404-isvc-687f8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw\" (UID: \"16b42b9f-fa74-4e57-8b69-f166541af5bd\") " pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"
May 20 15:54:03.977844 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.977615    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-687f8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41bf512d-9315-457b-a451-d7fb1eb230d4-success-200-isvc-687f8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-687f8-predictor-7d75c44d4d-64c45\" (UID: \"41bf512d-9315-457b-a451-d7fb1eb230d4\") " pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"
May 20 15:54:03.977844 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:54:03.977680    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-687f8-predictor-serving-cert: secret "success-200-isvc-687f8-predictor-serving-cert" not found
May 20 15:54:03.977844 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.977698    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l96ht\" (UniqueName: \"kubernetes.io/projected/41bf512d-9315-457b-a451-d7fb1eb230d4-kube-api-access-l96ht\") pod \"success-200-isvc-687f8-predictor-7d75c44d4d-64c45\" (UID: \"41bf512d-9315-457b-a451-d7fb1eb230d4\") " pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"
May 20 15:54:03.977844 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.977751    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16b42b9f-fa74-4e57-8b69-f166541af5bd-proxy-tls\") pod \"error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw\" (UID: \"16b42b9f-fa74-4e57-8b69-f166541af5bd\") " pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"
May 20 15:54:03.977844 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:54:03.977766    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41bf512d-9315-457b-a451-d7fb1eb230d4-proxy-tls podName:41bf512d-9315-457b-a451-d7fb1eb230d4 nodeName:}" failed. No retries permitted until 2026-05-20 15:54:04.477744162 +0000 UTC m=+1353.416585723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/41bf512d-9315-457b-a451-d7fb1eb230d4-proxy-tls") pod "success-200-isvc-687f8-predictor-7d75c44d4d-64c45" (UID: "41bf512d-9315-457b-a451-d7fb1eb230d4") : secret "success-200-isvc-687f8-predictor-serving-cert" not found
May 20 15:54:03.978391 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.978369    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-687f8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41bf512d-9315-457b-a451-d7fb1eb230d4-success-200-isvc-687f8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-687f8-predictor-7d75c44d4d-64c45\" (UID: \"41bf512d-9315-457b-a451-d7fb1eb230d4\") " pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"
May 20 15:54:03.993089 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:03.993012    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l96ht\" (UniqueName: \"kubernetes.io/projected/41bf512d-9315-457b-a451-d7fb1eb230d4-kube-api-access-l96ht\") pod \"success-200-isvc-687f8-predictor-7d75c44d4d-64c45\" (UID: \"41bf512d-9315-457b-a451-d7fb1eb230d4\") " pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"
May 20 15:54:04.078377 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:04.078343    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w6vwk\" (UniqueName: \"kubernetes.io/projected/16b42b9f-fa74-4e57-8b69-f166541af5bd-kube-api-access-w6vwk\") pod \"error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw\" (UID: \"16b42b9f-fa74-4e57-8b69-f166541af5bd\") " pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"
May 20 15:54:04.078592 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:04.078394    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-687f8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16b42b9f-fa74-4e57-8b69-f166541af5bd-error-404-isvc-687f8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw\" (UID: \"16b42b9f-fa74-4e57-8b69-f166541af5bd\") " pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"
May 20 15:54:04.078592 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:04.078440    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16b42b9f-fa74-4e57-8b69-f166541af5bd-proxy-tls\") pod \"error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw\" (UID: \"16b42b9f-fa74-4e57-8b69-f166541af5bd\") " pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"
May 20 15:54:04.079230 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:04.079161    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-687f8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16b42b9f-fa74-4e57-8b69-f166541af5bd-error-404-isvc-687f8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw\" (UID: \"16b42b9f-fa74-4e57-8b69-f166541af5bd\") " pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"
May 20 15:54:04.081071 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:04.081010    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16b42b9f-fa74-4e57-8b69-f166541af5bd-proxy-tls\") pod \"error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw\" (UID: \"16b42b9f-fa74-4e57-8b69-f166541af5bd\") " pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"
May 20 15:54:04.096429 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:04.096403    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6vwk\" (UniqueName: \"kubernetes.io/projected/16b42b9f-fa74-4e57-8b69-f166541af5bd-kube-api-access-w6vwk\") pod \"error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw\" (UID: \"16b42b9f-fa74-4e57-8b69-f166541af5bd\") " pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"
May 20 15:54:04.147205 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:04.147170    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"
May 20 15:54:04.282060 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:04.281669    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"]
May 20 15:54:04.283563 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:54:04.283536    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b42b9f_fa74_4e57_8b69_f166541af5bd.slice/crio-05ee8a1087adfeb849cb78b75222177c51206bdc2da1f4c828499a9c81d42c95 WatchSource:0}: Error finding container 05ee8a1087adfeb849cb78b75222177c51206bdc2da1f4c828499a9c81d42c95: Status 404 returned error can't find the container with id 05ee8a1087adfeb849cb78b75222177c51206bdc2da1f4c828499a9c81d42c95
May 20 15:54:04.285276 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:04.285259    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:54:04.358758 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:04.358723    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw" event={"ID":"16b42b9f-fa74-4e57-8b69-f166541af5bd","Type":"ContainerStarted","Data":"10b26717daa00dfc2c0f68332cd1723c95fadb40d133e855595eb44d1d43577d"}
May 20 15:54:04.358932 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:04.358772    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw" event={"ID":"16b42b9f-fa74-4e57-8b69-f166541af5bd","Type":"ContainerStarted","Data":"05ee8a1087adfeb849cb78b75222177c51206bdc2da1f4c828499a9c81d42c95"}
May 20 15:54:04.360396 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:04.360374    2568 generic.go:358] "Generic (PLEG): container finished" podID="1a538143-ec1a-49c3-9204-e35cf7adbd2e" containerID="71831e30ba3629096a4bf0f9630318884c003e44988600665015f88b87e3cc86" exitCode=2
May 20 15:54:04.360490 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:04.360445    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg" event={"ID":"1a538143-ec1a-49c3-9204-e35cf7adbd2e","Type":"ContainerDied","Data":"71831e30ba3629096a4bf0f9630318884c003e44988600665015f88b87e3cc86"}
May 20 15:54:04.362128 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:04.362101    2568 generic.go:358] "Generic (PLEG): container finished" podID="4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b" containerID="1bb6886bb8e627fbd06a2116f8278a7586f991d6b4e80df3bcd234e1738369ec" exitCode=2
May 20 15:54:04.362215 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:04.362171    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5" event={"ID":"4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b","Type":"ContainerDied","Data":"1bb6886bb8e627fbd06a2116f8278a7586f991d6b4e80df3bcd234e1738369ec"}
May 20 15:54:04.481600 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:04.481557    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41bf512d-9315-457b-a451-d7fb1eb230d4-proxy-tls\") pod \"success-200-isvc-687f8-predictor-7d75c44d4d-64c45\" (UID: \"41bf512d-9315-457b-a451-d7fb1eb230d4\") " pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"
May 20 15:54:04.484122 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:04.484103    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41bf512d-9315-457b-a451-d7fb1eb230d4-proxy-tls\") pod \"success-200-isvc-687f8-predictor-7d75c44d4d-64c45\" (UID: \"41bf512d-9315-457b-a451-d7fb1eb230d4\") " pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"
May 20 15:54:04.698730 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:04.698639    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"
May 20 15:54:04.832211 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:04.832181    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"]
May 20 15:54:04.834823 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:54:04.834791    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41bf512d_9315_457b_a451_d7fb1eb230d4.slice/crio-466fb5601ba7301b6dec9ab9352bcf5b429a76f60dd989fc75c4d063d683887f WatchSource:0}: Error finding container 466fb5601ba7301b6dec9ab9352bcf5b429a76f60dd989fc75c4d063d683887f: Status 404 returned error can't find the container with id 466fb5601ba7301b6dec9ab9352bcf5b429a76f60dd989fc75c4d063d683887f
May 20 15:54:05.366974 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:05.366942    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw" event={"ID":"16b42b9f-fa74-4e57-8b69-f166541af5bd","Type":"ContainerStarted","Data":"f89266cb60c15bb10e64d197cd7b8fdad39b4cab9c4db78c8a2e4e279d20e097"}
May 20 15:54:05.367268 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:05.367231    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"
May 20 15:54:05.367268 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:05.367269    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"
May 20 15:54:05.368981 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:05.368734    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw" podUID="16b42b9f-fa74-4e57-8b69-f166541af5bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
May 20 15:54:05.368981 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:05.368785    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45" event={"ID":"41bf512d-9315-457b-a451-d7fb1eb230d4","Type":"ContainerStarted","Data":"81ceaa5ac9b87b883e20fca58d954bdac96e9821af2755ab576bff7a0a476fef"}
May 20 15:54:05.368981 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:05.368813    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45" event={"ID":"41bf512d-9315-457b-a451-d7fb1eb230d4","Type":"ContainerStarted","Data":"8180acaa5a3968e0e3528da7b66576021e423ee45536140450de254427187960"}
May 20 15:54:05.368981 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:05.368828    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45" event={"ID":"41bf512d-9315-457b-a451-d7fb1eb230d4","Type":"ContainerStarted","Data":"466fb5601ba7301b6dec9ab9352bcf5b429a76f60dd989fc75c4d063d683887f"}
May 20 15:54:05.368981 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:05.368936    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"
May 20 15:54:05.368981 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:05.368974    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"
May 20 15:54:05.369896 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:05.369867    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45" podUID="41bf512d-9315-457b-a451-d7fb1eb230d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 20 15:54:05.385713 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:05.385661    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw" podStartSLOduration=2.385645246 podStartE2EDuration="2.385645246s" podCreationTimestamp="2026-05-20 15:54:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:54:05.383830769 +0000 UTC m=+1354.322672327" watchObservedRunningTime="2026-05-20 15:54:05.385645246 +0000 UTC m=+1354.324486804"
May 20 15:54:05.401737 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:05.401694    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45" podStartSLOduration=2.401679765 podStartE2EDuration="2.401679765s" podCreationTimestamp="2026-05-20 15:54:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:54:05.399404674 +0000 UTC m=+1354.338246232" watchObservedRunningTime="2026-05-20 15:54:05.401679765 +0000 UTC m=+1354.340521321"
May 20 15:54:05.608189 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:05.608144    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c" podUID="82ba7204-c40a-4fdc-97de-6257e21c8155" containerName="switch-graph-d4d66" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:54:06.325357 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:06.325312    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg" podUID="1a538143-ec1a-49c3-9204-e35cf7adbd2e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
May 20 15:54:06.330285 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:06.330243    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg" podUID="1a538143-ec1a-49c3-9204-e35cf7adbd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 20 15:54:06.373369 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:06.373319    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45" podUID="41bf512d-9315-457b-a451-d7fb1eb230d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 20 15:54:06.373612 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:06.373340    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw" podUID="16b42b9f-fa74-4e57-8b69-f166541af5bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
May 20 15:54:07.115474 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.115450    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"
May 20 15:54:07.206642 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.206609    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-d4d66-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b-error-404-isvc-d4d66-kube-rbac-proxy-sar-config\") pod \"4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b\" (UID: \"4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b\") "
May 20 15:54:07.206785 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.206664    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b-proxy-tls\") pod \"4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b\" (UID: \"4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b\") "
May 20 15:54:07.206785 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.206710    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvbz7\" (UniqueName: \"kubernetes.io/projected/4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b-kube-api-access-jvbz7\") pod \"4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b\" (UID: \"4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b\") "
May 20 15:54:07.207038 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.206989    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b-error-404-isvc-d4d66-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-d4d66-kube-rbac-proxy-sar-config") pod "4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b" (UID: "4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b"). InnerVolumeSpecName "error-404-isvc-d4d66-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:54:07.208915 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.208895    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b-kube-api-access-jvbz7" (OuterVolumeSpecName: "kube-api-access-jvbz7") pod "4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b" (UID: "4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b"). InnerVolumeSpecName "kube-api-access-jvbz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:54:07.208978 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.208921    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b" (UID: "4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:54:07.307688 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.307653    2568 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-d4d66-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b-error-404-isvc-d4d66-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:54:07.307688 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.307683    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:54:07.307858 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.307696    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvbz7\" (UniqueName: \"kubernetes.io/projected/4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b-kube-api-access-jvbz7\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:54:07.360920 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.360893    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"
May 20 15:54:07.377522 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.377489    2568 generic.go:358] "Generic (PLEG): container finished" podID="1a538143-ec1a-49c3-9204-e35cf7adbd2e" containerID="efdfbb1c5bae91bb24aae5b5868374fd18cd8b110a0d828936f0e9a1c11ca75c" exitCode=0
May 20 15:54:07.377671 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.377565    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg" event={"ID":"1a538143-ec1a-49c3-9204-e35cf7adbd2e","Type":"ContainerDied","Data":"efdfbb1c5bae91bb24aae5b5868374fd18cd8b110a0d828936f0e9a1c11ca75c"}
May 20 15:54:07.377671 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.377586    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"
May 20 15:54:07.377671 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.377603    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg" event={"ID":"1a538143-ec1a-49c3-9204-e35cf7adbd2e","Type":"ContainerDied","Data":"c1249ad428d4a1d204ce2627e94c0c068d51912306a126f72bf5bfab7824569f"}
May 20 15:54:07.377671 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.377618    2568 scope.go:117] "RemoveContainer" containerID="71831e30ba3629096a4bf0f9630318884c003e44988600665015f88b87e3cc86"
May 20 15:54:07.379785 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.379730    2568 generic.go:358] "Generic (PLEG): container finished" podID="4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b" containerID="cf92198c9d9f5e9e63795e4e62194f097be5b09cec181c81a8521944b6b294c6" exitCode=0
May 20 15:54:07.379911 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.379881    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"
May 20 15:54:07.380136 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.380109    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5" event={"ID":"4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b","Type":"ContainerDied","Data":"cf92198c9d9f5e9e63795e4e62194f097be5b09cec181c81a8521944b6b294c6"}
May 20 15:54:07.380281 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.380149    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5" event={"ID":"4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b","Type":"ContainerDied","Data":"b1255f5852df0557555b34f02a6cc4a7f155af2831a95acbbc20241609ff094e"}
May 20 15:54:07.388524 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.388505    2568 scope.go:117] "RemoveContainer" containerID="efdfbb1c5bae91bb24aae5b5868374fd18cd8b110a0d828936f0e9a1c11ca75c"
May 20 15:54:07.397012 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.396994    2568 scope.go:117] "RemoveContainer" containerID="71831e30ba3629096a4bf0f9630318884c003e44988600665015f88b87e3cc86"
May 20 15:54:07.397452 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:54:07.397425    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71831e30ba3629096a4bf0f9630318884c003e44988600665015f88b87e3cc86\": container with ID starting with 71831e30ba3629096a4bf0f9630318884c003e44988600665015f88b87e3cc86 not found: ID does not exist" containerID="71831e30ba3629096a4bf0f9630318884c003e44988600665015f88b87e3cc86"
May 20 15:54:07.397538 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.397460    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71831e30ba3629096a4bf0f9630318884c003e44988600665015f88b87e3cc86"} err="failed to get container status \"71831e30ba3629096a4bf0f9630318884c003e44988600665015f88b87e3cc86\": rpc error: code = NotFound desc = could not find container \"71831e30ba3629096a4bf0f9630318884c003e44988600665015f88b87e3cc86\": container with ID starting with 71831e30ba3629096a4bf0f9630318884c003e44988600665015f88b87e3cc86 not found: ID does not exist"
May 20 15:54:07.397538 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.397482    2568 scope.go:117] "RemoveContainer" containerID="efdfbb1c5bae91bb24aae5b5868374fd18cd8b110a0d828936f0e9a1c11ca75c"
May 20 15:54:07.397695 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:54:07.397681    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efdfbb1c5bae91bb24aae5b5868374fd18cd8b110a0d828936f0e9a1c11ca75c\": container with ID starting with efdfbb1c5bae91bb24aae5b5868374fd18cd8b110a0d828936f0e9a1c11ca75c not found: ID does not exist" containerID="efdfbb1c5bae91bb24aae5b5868374fd18cd8b110a0d828936f0e9a1c11ca75c"
May 20 15:54:07.397740 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.397699    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efdfbb1c5bae91bb24aae5b5868374fd18cd8b110a0d828936f0e9a1c11ca75c"} err="failed to get container status \"efdfbb1c5bae91bb24aae5b5868374fd18cd8b110a0d828936f0e9a1c11ca75c\": rpc error: code = NotFound desc = could not find container \"efdfbb1c5bae91bb24aae5b5868374fd18cd8b110a0d828936f0e9a1c11ca75c\": container with ID starting with efdfbb1c5bae91bb24aae5b5868374fd18cd8b110a0d828936f0e9a1c11ca75c not found: ID does not exist"
May 20 15:54:07.397740 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.397712    2568 scope.go:117] "RemoveContainer" containerID="1bb6886bb8e627fbd06a2116f8278a7586f991d6b4e80df3bcd234e1738369ec"
May 20 15:54:07.408703 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.408649    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"]
May 20 15:54:07.410662 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.410620    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d4d66-predictor-867b8b47f7-js4z5"]
May 20 15:54:07.412100 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.411164    2568 scope.go:117] "RemoveContainer" containerID="cf92198c9d9f5e9e63795e4e62194f097be5b09cec181c81a8521944b6b294c6"
May 20 15:54:07.419967 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.419948    2568 scope.go:117] "RemoveContainer" containerID="1bb6886bb8e627fbd06a2116f8278a7586f991d6b4e80df3bcd234e1738369ec"
May 20 15:54:07.420282 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:54:07.420260    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb6886bb8e627fbd06a2116f8278a7586f991d6b4e80df3bcd234e1738369ec\": container with ID starting with 1bb6886bb8e627fbd06a2116f8278a7586f991d6b4e80df3bcd234e1738369ec not found: ID does not exist" containerID="1bb6886bb8e627fbd06a2116f8278a7586f991d6b4e80df3bcd234e1738369ec"
May 20 15:54:07.420349 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.420290    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb6886bb8e627fbd06a2116f8278a7586f991d6b4e80df3bcd234e1738369ec"} err="failed to get container status \"1bb6886bb8e627fbd06a2116f8278a7586f991d6b4e80df3bcd234e1738369ec\": rpc error: code = NotFound desc = could not find container \"1bb6886bb8e627fbd06a2116f8278a7586f991d6b4e80df3bcd234e1738369ec\": container with ID starting with 1bb6886bb8e627fbd06a2116f8278a7586f991d6b4e80df3bcd234e1738369ec not found: ID does not exist"
May 20 15:54:07.420349 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.420308    2568 scope.go:117] "RemoveContainer" containerID="cf92198c9d9f5e9e63795e4e62194f097be5b09cec181c81a8521944b6b294c6"
May 20 15:54:07.420550 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:54:07.420533    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf92198c9d9f5e9e63795e4e62194f097be5b09cec181c81a8521944b6b294c6\": container with ID starting with cf92198c9d9f5e9e63795e4e62194f097be5b09cec181c81a8521944b6b294c6 not found: ID does not exist" containerID="cf92198c9d9f5e9e63795e4e62194f097be5b09cec181c81a8521944b6b294c6"
May 20 15:54:07.420584 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.420555    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf92198c9d9f5e9e63795e4e62194f097be5b09cec181c81a8521944b6b294c6"} err="failed to get container status \"cf92198c9d9f5e9e63795e4e62194f097be5b09cec181c81a8521944b6b294c6\": rpc error: code = NotFound desc = could not find container \"cf92198c9d9f5e9e63795e4e62194f097be5b09cec181c81a8521944b6b294c6\": container with ID starting with cf92198c9d9f5e9e63795e4e62194f097be5b09cec181c81a8521944b6b294c6 not found: ID does not exist"
May 20 15:54:07.510642 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.510559    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a538143-ec1a-49c3-9204-e35cf7adbd2e-proxy-tls\") pod \"1a538143-ec1a-49c3-9204-e35cf7adbd2e\" (UID: \"1a538143-ec1a-49c3-9204-e35cf7adbd2e\") "
May 20 15:54:07.510642 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.510592    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8vf5\" (UniqueName: \"kubernetes.io/projected/1a538143-ec1a-49c3-9204-e35cf7adbd2e-kube-api-access-s8vf5\") pod \"1a538143-ec1a-49c3-9204-e35cf7adbd2e\" (UID: \"1a538143-ec1a-49c3-9204-e35cf7adbd2e\") "
May 20 15:54:07.510642 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.510620    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-d4d66-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a538143-ec1a-49c3-9204-e35cf7adbd2e-success-200-isvc-d4d66-kube-rbac-proxy-sar-config\") pod \"1a538143-ec1a-49c3-9204-e35cf7adbd2e\" (UID: \"1a538143-ec1a-49c3-9204-e35cf7adbd2e\") "
May 20 15:54:07.511078 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.511051    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a538143-ec1a-49c3-9204-e35cf7adbd2e-success-200-isvc-d4d66-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-d4d66-kube-rbac-proxy-sar-config") pod "1a538143-ec1a-49c3-9204-e35cf7adbd2e" (UID: "1a538143-ec1a-49c3-9204-e35cf7adbd2e"). InnerVolumeSpecName "success-200-isvc-d4d66-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:54:07.512906 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.512885    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a538143-ec1a-49c3-9204-e35cf7adbd2e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a538143-ec1a-49c3-9204-e35cf7adbd2e" (UID: "1a538143-ec1a-49c3-9204-e35cf7adbd2e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:54:07.512977 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.512921    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a538143-ec1a-49c3-9204-e35cf7adbd2e-kube-api-access-s8vf5" (OuterVolumeSpecName: "kube-api-access-s8vf5") pod "1a538143-ec1a-49c3-9204-e35cf7adbd2e" (UID: "1a538143-ec1a-49c3-9204-e35cf7adbd2e"). InnerVolumeSpecName "kube-api-access-s8vf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:54:07.611297 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.611258    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a538143-ec1a-49c3-9204-e35cf7adbd2e-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:54:07.611297 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.611291    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8vf5\" (UniqueName: \"kubernetes.io/projected/1a538143-ec1a-49c3-9204-e35cf7adbd2e-kube-api-access-s8vf5\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:54:07.611297 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.611302    2568 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-d4d66-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a538143-ec1a-49c3-9204-e35cf7adbd2e-success-200-isvc-d4d66-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:54:07.643835 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.643805    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b" path="/var/lib/kubelet/pods/4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b/volumes"
May 20 15:54:07.697490 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.697459    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"]
May 20 15:54:07.700626 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:07.700596    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d4d66-predictor-54ddf55d4b-67mdg"]
May 20 15:54:09.643793 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:09.643756    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a538143-ec1a-49c3-9204-e35cf7adbd2e" path="/var/lib/kubelet/pods/1a538143-ec1a-49c3-9204-e35cf7adbd2e/volumes"
May 20 15:54:10.608184 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:10.608146    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c" podUID="82ba7204-c40a-4fdc-97de-6257e21c8155" containerName="switch-graph-d4d66" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:54:11.378387 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:11.378359    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"
May 20 15:54:11.378902 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:11.378879    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"
May 20 15:54:11.378974 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:11.378941    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45" podUID="41bf512d-9315-457b-a451-d7fb1eb230d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 20 15:54:11.379306 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:11.379283    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw" podUID="16b42b9f-fa74-4e57-8b69-f166541af5bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
May 20 15:54:15.608261 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:15.608218    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c" podUID="82ba7204-c40a-4fdc-97de-6257e21c8155" containerName="switch-graph-d4d66" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:54:15.608708 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:15.608386    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c"
May 20 15:54:20.608008 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:20.607965    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c" podUID="82ba7204-c40a-4fdc-97de-6257e21c8155" containerName="switch-graph-d4d66" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:54:21.379685 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:21.379651    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45" podUID="41bf512d-9315-457b-a451-d7fb1eb230d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 20 15:54:21.379852 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:21.379651    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw" podUID="16b42b9f-fa74-4e57-8b69-f166541af5bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
May 20 15:54:25.607958 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:25.607907    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c" podUID="82ba7204-c40a-4fdc-97de-6257e21c8155" containerName="switch-graph-d4d66" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:54:29.310809 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.310773    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29"]
May 20 15:54:29.311299 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.311101    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29" podUID="7d23dde2-c954-42ec-84b2-2686d2e578ad" containerName="sequence-graph-23504" containerID="cri-o://4045ca7d46594da939e0517d0d54e65b5485819637f0daa4004094e9e1d164ef" gracePeriod=30
May 20 15:54:29.407198 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.407160    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"]
May 20 15:54:29.407551 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.407517    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7" podUID="7922d8ef-19c4-4ef9-a09b-f16db6cf33e5" containerName="kserve-container" containerID="cri-o://19dcbe5fb0f248c40b43620817e47f97fb47fd189e087afa1e8b252727df2f85" gracePeriod=30
May 20 15:54:29.407655 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.407566    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7" podUID="7922d8ef-19c4-4ef9-a09b-f16db6cf33e5" containerName="kube-rbac-proxy" containerID="cri-o://37ff1e4e649c408617b071e004fb1e8c7799cf02d8fba1857d75ae7cf9520ae0" gracePeriod=30
May 20 15:54:29.472457 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.472401    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"]
May 20 15:54:29.473424 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.473383    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a538143-ec1a-49c3-9204-e35cf7adbd2e" containerName="kserve-container"
May 20 15:54:29.473424 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.473413    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a538143-ec1a-49c3-9204-e35cf7adbd2e" containerName="kserve-container"
May 20 15:54:29.473424 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.473427    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a538143-ec1a-49c3-9204-e35cf7adbd2e" containerName="kube-rbac-proxy"
May 20 15:54:29.473640 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.473435    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a538143-ec1a-49c3-9204-e35cf7adbd2e" containerName="kube-rbac-proxy"
May 20 15:54:29.473640 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.473451    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b" containerName="kserve-container"
May 20 15:54:29.473640 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.473459    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b" containerName="kserve-container"
May 20 15:54:29.473640 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.473485    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b" containerName="kube-rbac-proxy"
May 20 15:54:29.473640 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.473494    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b" containerName="kube-rbac-proxy"
May 20 15:54:29.473640 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.473592    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b" containerName="kube-rbac-proxy"
May 20 15:54:29.473640 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.473607    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="4aa54823-53e0-4d1e-aafe-9c7f98cc3c0b" containerName="kserve-container"
May 20 15:54:29.473640 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.473618    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="1a538143-ec1a-49c3-9204-e35cf7adbd2e" containerName="kserve-container"
May 20 15:54:29.473640 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.473629    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="1a538143-ec1a-49c3-9204-e35cf7adbd2e" containerName="kube-rbac-proxy"
May 20 15:54:29.485604 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.485567    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"]
May 20 15:54:29.486598 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.485762    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"
May 20 15:54:29.486697 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.486549    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6" podUID="45e8ec2e-cb37-46fd-9bca-db9bcd707e33" containerName="kserve-container" containerID="cri-o://74c2486c69dab3b26ad1e6fbb2b88bfbeb1abdb9660e118e1fce209ec41451de" gracePeriod=30
May 20 15:54:29.486922 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.486875    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6" podUID="45e8ec2e-cb37-46fd-9bca-db9bcd707e33" containerName="kube-rbac-proxy" containerID="cri-o://ac3a54b558db5fcef62aab9c24890ce2233032a8ba2eca8f7aa73297fe90924c" gracePeriod=30
May 20 15:54:29.489374 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.489122    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4ff76-predictor-serving-cert\""
May 20 15:54:29.489374 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.489147    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4ff76-kube-rbac-proxy-sar-config\""
May 20 15:54:29.489374 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.489310    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"]
May 20 15:54:29.580802 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.580713    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"]
May 20 15:54:29.584375 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.584350    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"
May 20 15:54:29.586432 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.586407    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4ff76-predictor-serving-cert\""
May 20 15:54:29.586598 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.586582    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4ff76-kube-rbac-proxy-sar-config\""
May 20 15:54:29.595285 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.595243    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"]
May 20 15:54:29.600726 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.600694    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-4ff76-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbffac89-efb6-4741-b082-16c3d625f6b5-success-200-isvc-4ff76-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4ff76-predictor-5878d445d4-msxvp\" (UID: \"bbffac89-efb6-4741-b082-16c3d625f6b5\") " pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"
May 20 15:54:29.600889 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.600764    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbffac89-efb6-4741-b082-16c3d625f6b5-proxy-tls\") pod \"success-200-isvc-4ff76-predictor-5878d445d4-msxvp\" (UID: \"bbffac89-efb6-4741-b082-16c3d625f6b5\") " pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"
May 20 15:54:29.600889 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.600812    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxzt6\" (UniqueName: \"kubernetes.io/projected/bbffac89-efb6-4741-b082-16c3d625f6b5-kube-api-access-dxzt6\") pod \"success-200-isvc-4ff76-predictor-5878d445d4-msxvp\" (UID: \"bbffac89-efb6-4741-b082-16c3d625f6b5\") " pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"
May 20 15:54:29.701359 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.701318    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da04dcf0-5363-4b02-b5ba-8e39acd77721-proxy-tls\") pod \"error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp\" (UID: \"da04dcf0-5363-4b02-b5ba-8e39acd77721\") " pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"
May 20 15:54:29.701571 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.701374    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbffac89-efb6-4741-b082-16c3d625f6b5-proxy-tls\") pod \"success-200-isvc-4ff76-predictor-5878d445d4-msxvp\" (UID: \"bbffac89-efb6-4741-b082-16c3d625f6b5\") " pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"
May 20 15:54:29.701571 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.701437    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dxzt6\" (UniqueName: \"kubernetes.io/projected/bbffac89-efb6-4741-b082-16c3d625f6b5-kube-api-access-dxzt6\") pod \"success-200-isvc-4ff76-predictor-5878d445d4-msxvp\" (UID: \"bbffac89-efb6-4741-b082-16c3d625f6b5\") " pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"
May 20 15:54:29.701571 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.701500    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-4ff76-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da04dcf0-5363-4b02-b5ba-8e39acd77721-error-404-isvc-4ff76-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp\" (UID: \"da04dcf0-5363-4b02-b5ba-8e39acd77721\") " pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"
May 20 15:54:29.701571 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:54:29.701503    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-serving-cert: secret "success-200-isvc-4ff76-predictor-serving-cert" not found
May 20 15:54:29.701571 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.701528    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz7rl\" (UniqueName: \"kubernetes.io/projected/da04dcf0-5363-4b02-b5ba-8e39acd77721-kube-api-access-sz7rl\") pod \"error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp\" (UID: \"da04dcf0-5363-4b02-b5ba-8e39acd77721\") " pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"
May 20 15:54:29.701804 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.701573    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-4ff76-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbffac89-efb6-4741-b082-16c3d625f6b5-success-200-isvc-4ff76-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4ff76-predictor-5878d445d4-msxvp\" (UID: \"bbffac89-efb6-4741-b082-16c3d625f6b5\") " pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"
May 20 15:54:29.701804 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:54:29.701608    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbffac89-efb6-4741-b082-16c3d625f6b5-proxy-tls podName:bbffac89-efb6-4741-b082-16c3d625f6b5 nodeName:}" failed. No retries permitted until 2026-05-20 15:54:30.201584724 +0000 UTC m=+1379.140426258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bbffac89-efb6-4741-b082-16c3d625f6b5-proxy-tls") pod "success-200-isvc-4ff76-predictor-5878d445d4-msxvp" (UID: "bbffac89-efb6-4741-b082-16c3d625f6b5") : secret "success-200-isvc-4ff76-predictor-serving-cert" not found
May 20 15:54:29.702299 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.702276    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-4ff76-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbffac89-efb6-4741-b082-16c3d625f6b5-success-200-isvc-4ff76-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4ff76-predictor-5878d445d4-msxvp\" (UID: \"bbffac89-efb6-4741-b082-16c3d625f6b5\") " pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"
May 20 15:54:29.710604 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.710577    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxzt6\" (UniqueName: \"kubernetes.io/projected/bbffac89-efb6-4741-b082-16c3d625f6b5-kube-api-access-dxzt6\") pod \"success-200-isvc-4ff76-predictor-5878d445d4-msxvp\" (UID: \"bbffac89-efb6-4741-b082-16c3d625f6b5\") " pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"
May 20 15:54:29.803004 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.802955    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-4ff76-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da04dcf0-5363-4b02-b5ba-8e39acd77721-error-404-isvc-4ff76-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp\" (UID: \"da04dcf0-5363-4b02-b5ba-8e39acd77721\") " pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"
May 20 15:54:29.803004 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.803008    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sz7rl\" (UniqueName: \"kubernetes.io/projected/da04dcf0-5363-4b02-b5ba-8e39acd77721-kube-api-access-sz7rl\") pod \"error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp\" (UID: \"da04dcf0-5363-4b02-b5ba-8e39acd77721\") " pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"
May 20 15:54:29.803300 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.803098    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da04dcf0-5363-4b02-b5ba-8e39acd77721-proxy-tls\") pod \"error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp\" (UID: \"da04dcf0-5363-4b02-b5ba-8e39acd77721\") " pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"
May 20 15:54:29.803717 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.803690    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-4ff76-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da04dcf0-5363-4b02-b5ba-8e39acd77721-error-404-isvc-4ff76-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp\" (UID: \"da04dcf0-5363-4b02-b5ba-8e39acd77721\") " pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"
May 20 15:54:29.805743 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.805718    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da04dcf0-5363-4b02-b5ba-8e39acd77721-proxy-tls\") pod \"error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp\" (UID: \"da04dcf0-5363-4b02-b5ba-8e39acd77721\") " pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"
May 20 15:54:29.811065 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.811009    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz7rl\" (UniqueName: \"kubernetes.io/projected/da04dcf0-5363-4b02-b5ba-8e39acd77721-kube-api-access-sz7rl\") pod \"error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp\" (UID: \"da04dcf0-5363-4b02-b5ba-8e39acd77721\") " pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"
May 20 15:54:29.897324 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:29.897232    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"
May 20 15:54:30.045714 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:30.045689    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"]
May 20 15:54:30.047408 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:54:30.047384    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice/crio-1079a8636ab0dd4e7f908c93c11547e5674c8c1859385170c3283cd6765f2f99 WatchSource:0}: Error finding container 1079a8636ab0dd4e7f908c93c11547e5674c8c1859385170c3283cd6765f2f99: Status 404 returned error can't find the container with id 1079a8636ab0dd4e7f908c93c11547e5674c8c1859385170c3283cd6765f2f99
May 20 15:54:30.209133 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:30.209095    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbffac89-efb6-4741-b082-16c3d625f6b5-proxy-tls\") pod \"success-200-isvc-4ff76-predictor-5878d445d4-msxvp\" (UID: \"bbffac89-efb6-4741-b082-16c3d625f6b5\") " pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"
May 20 15:54:30.211632 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:30.211610    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbffac89-efb6-4741-b082-16c3d625f6b5-proxy-tls\") pod \"success-200-isvc-4ff76-predictor-5878d445d4-msxvp\" (UID: \"bbffac89-efb6-4741-b082-16c3d625f6b5\") " pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"
May 20 15:54:30.414852 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:30.414761    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"
May 20 15:54:30.464032 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:30.463985    2568 generic.go:358] "Generic (PLEG): container finished" podID="7922d8ef-19c4-4ef9-a09b-f16db6cf33e5" containerID="37ff1e4e649c408617b071e004fb1e8c7799cf02d8fba1857d75ae7cf9520ae0" exitCode=2
May 20 15:54:30.464223 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:30.464052    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7" event={"ID":"7922d8ef-19c4-4ef9-a09b-f16db6cf33e5","Type":"ContainerDied","Data":"37ff1e4e649c408617b071e004fb1e8c7799cf02d8fba1857d75ae7cf9520ae0"}
May 20 15:54:30.466229 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:30.465902    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp" event={"ID":"da04dcf0-5363-4b02-b5ba-8e39acd77721","Type":"ContainerStarted","Data":"386540fc48ed1cac7db40815b16a84a49147126586b32bdfa9e3d0077746b9ac"}
May 20 15:54:30.466229 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:30.465936    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp" event={"ID":"da04dcf0-5363-4b02-b5ba-8e39acd77721","Type":"ContainerStarted","Data":"506926f355d1a0728dc934e0627346ee68a2a0ebb59fb22f3d2e2441dc4b6e62"}
May 20 15:54:30.466229 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:30.465950    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp" event={"ID":"da04dcf0-5363-4b02-b5ba-8e39acd77721","Type":"ContainerStarted","Data":"1079a8636ab0dd4e7f908c93c11547e5674c8c1859385170c3283cd6765f2f99"}
May 20 15:54:30.466229 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:30.466063    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"
May 20 15:54:30.472145 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:30.472051    2568 generic.go:358] "Generic (PLEG): container finished" podID="45e8ec2e-cb37-46fd-9bca-db9bcd707e33" containerID="ac3a54b558db5fcef62aab9c24890ce2233032a8ba2eca8f7aa73297fe90924c" exitCode=2
May 20 15:54:30.472145 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:30.472105    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6" event={"ID":"45e8ec2e-cb37-46fd-9bca-db9bcd707e33","Type":"ContainerDied","Data":"ac3a54b558db5fcef62aab9c24890ce2233032a8ba2eca8f7aa73297fe90924c"}
May 20 15:54:30.491085 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:30.491009    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp" podStartSLOduration=1.490992909 podStartE2EDuration="1.490992909s" podCreationTimestamp="2026-05-20 15:54:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:54:30.488781243 +0000 UTC m=+1379.427622800" watchObservedRunningTime="2026-05-20 15:54:30.490992909 +0000 UTC m=+1379.429834465"
May 20 15:54:30.570652 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:30.570585    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"]
May 20 15:54:30.575765 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:54:30.575714    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbffac89_efb6_4741_b082_16c3d625f6b5.slice/crio-dbfde16948983f994b00708539d4e51eb60d5b651c1c460fb26311920148aac0 WatchSource:0}: Error finding container dbfde16948983f994b00708539d4e51eb60d5b651c1c460fb26311920148aac0: Status 404 returned error can't find the container with id dbfde16948983f994b00708539d4e51eb60d5b651c1c460fb26311920148aac0
May 20 15:54:30.608358 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:30.608325    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c" podUID="82ba7204-c40a-4fdc-97de-6257e21c8155" containerName="switch-graph-d4d66" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:54:31.379586 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:31.379538    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45" podUID="41bf512d-9315-457b-a451-d7fb1eb230d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 20 15:54:31.379586 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:31.379550    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw" podUID="16b42b9f-fa74-4e57-8b69-f166541af5bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
May 20 15:54:31.479108 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:31.479068    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp" event={"ID":"bbffac89-efb6-4741-b082-16c3d625f6b5","Type":"ContainerStarted","Data":"4b8434e7f16fff2c150f1c9a573152204aa35dffb09cf70c47631c74d644da14"}
May 20 15:54:31.479527 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:31.479118    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp" event={"ID":"bbffac89-efb6-4741-b082-16c3d625f6b5","Type":"ContainerStarted","Data":"4f9120b1e442a28a473b609246125b45c10b0642249bede65364223e67ee8b5b"}
May 20 15:54:31.479527 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:31.479138    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp" event={"ID":"bbffac89-efb6-4741-b082-16c3d625f6b5","Type":"ContainerStarted","Data":"dbfde16948983f994b00708539d4e51eb60d5b651c1c460fb26311920148aac0"}
May 20 15:54:31.479527 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:31.479221    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"
May 20 15:54:31.479527 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:31.479416    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"
May 20 15:54:31.480934 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:31.480905    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp" podUID="da04dcf0-5363-4b02-b5ba-8e39acd77721" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
May 20 15:54:31.499341 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:31.499286    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp" podStartSLOduration=2.499268637 podStartE2EDuration="2.499268637s" podCreationTimestamp="2026-05-20 15:54:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:54:31.496903303 +0000 UTC m=+1380.435744860" watchObservedRunningTime="2026-05-20 15:54:31.499268637 +0000 UTC m=+1380.438110194"
May 20 15:54:31.699092 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:31.698985    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29" podUID="7d23dde2-c954-42ec-84b2-2686d2e578ad" containerName="sequence-graph-23504" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:54:32.475176 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:32.475127    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6" podUID="45e8ec2e-cb37-46fd-9bca-db9bcd707e33" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
May 20 15:54:32.481368 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:32.481335    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6" podUID="45e8ec2e-cb37-46fd-9bca-db9bcd707e33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
May 20 15:54:32.483912 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:32.483872    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp" podUID="da04dcf0-5363-4b02-b5ba-8e39acd77721" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
May 20 15:54:32.484104 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:32.484075    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"
May 20 15:54:32.485310 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:32.485285    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp" podUID="bbffac89-efb6-4741-b082-16c3d625f6b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
May 20 15:54:33.275793 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.275764    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"
May 20 15:54:33.337056 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.336935    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-23504-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7922d8ef-19c4-4ef9-a09b-f16db6cf33e5-success-200-isvc-23504-kube-rbac-proxy-sar-config\") pod \"7922d8ef-19c4-4ef9-a09b-f16db6cf33e5\" (UID: \"7922d8ef-19c4-4ef9-a09b-f16db6cf33e5\") "
May 20 15:54:33.337056 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.337041    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7922d8ef-19c4-4ef9-a09b-f16db6cf33e5-proxy-tls\") pod \"7922d8ef-19c4-4ef9-a09b-f16db6cf33e5\" (UID: \"7922d8ef-19c4-4ef9-a09b-f16db6cf33e5\") "
May 20 15:54:33.337315 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.337075    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mffcb\" (UniqueName: \"kubernetes.io/projected/7922d8ef-19c4-4ef9-a09b-f16db6cf33e5-kube-api-access-mffcb\") pod \"7922d8ef-19c4-4ef9-a09b-f16db6cf33e5\" (UID: \"7922d8ef-19c4-4ef9-a09b-f16db6cf33e5\") "
May 20 15:54:33.337477 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.337407    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7922d8ef-19c4-4ef9-a09b-f16db6cf33e5-success-200-isvc-23504-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-23504-kube-rbac-proxy-sar-config") pod "7922d8ef-19c4-4ef9-a09b-f16db6cf33e5" (UID: "7922d8ef-19c4-4ef9-a09b-f16db6cf33e5"). InnerVolumeSpecName "success-200-isvc-23504-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:54:33.339319 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.339291    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7922d8ef-19c4-4ef9-a09b-f16db6cf33e5-kube-api-access-mffcb" (OuterVolumeSpecName: "kube-api-access-mffcb") pod "7922d8ef-19c4-4ef9-a09b-f16db6cf33e5" (UID: "7922d8ef-19c4-4ef9-a09b-f16db6cf33e5"). InnerVolumeSpecName "kube-api-access-mffcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:54:33.339319 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.339300    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7922d8ef-19c4-4ef9-a09b-f16db6cf33e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7922d8ef-19c4-4ef9-a09b-f16db6cf33e5" (UID: "7922d8ef-19c4-4ef9-a09b-f16db6cf33e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:54:33.438012 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.437969    2568 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-23504-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7922d8ef-19c4-4ef9-a09b-f16db6cf33e5-success-200-isvc-23504-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:54:33.438012 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.438004    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7922d8ef-19c4-4ef9-a09b-f16db6cf33e5-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:54:33.438012 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.438014    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mffcb\" (UniqueName: \"kubernetes.io/projected/7922d8ef-19c4-4ef9-a09b-f16db6cf33e5-kube-api-access-mffcb\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:54:33.488255 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.488218    2568 generic.go:358] "Generic (PLEG): container finished" podID="7922d8ef-19c4-4ef9-a09b-f16db6cf33e5" containerID="19dcbe5fb0f248c40b43620817e47f97fb47fd189e087afa1e8b252727df2f85" exitCode=0
May 20 15:54:33.488630 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.488302    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"
May 20 15:54:33.488630 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.488296    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7" event={"ID":"7922d8ef-19c4-4ef9-a09b-f16db6cf33e5","Type":"ContainerDied","Data":"19dcbe5fb0f248c40b43620817e47f97fb47fd189e087afa1e8b252727df2f85"}
May 20 15:54:33.488630 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.488401    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7" event={"ID":"7922d8ef-19c4-4ef9-a09b-f16db6cf33e5","Type":"ContainerDied","Data":"8c323eac01dd3d7ea2e3313f2b10e69f304fb8a23a1f485804a4df02d7219eb4"}
May 20 15:54:33.488630 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.488416    2568 scope.go:117] "RemoveContainer" containerID="37ff1e4e649c408617b071e004fb1e8c7799cf02d8fba1857d75ae7cf9520ae0"
May 20 15:54:33.488981 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.488931    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp" podUID="bbffac89-efb6-4741-b082-16c3d625f6b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
May 20 15:54:33.497263 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.497246    2568 scope.go:117] "RemoveContainer" containerID="19dcbe5fb0f248c40b43620817e47f97fb47fd189e087afa1e8b252727df2f85"
May 20 15:54:33.504920 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.504904    2568 scope.go:117] "RemoveContainer" containerID="37ff1e4e649c408617b071e004fb1e8c7799cf02d8fba1857d75ae7cf9520ae0"
May 20 15:54:33.505221 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:54:33.505201    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ff1e4e649c408617b071e004fb1e8c7799cf02d8fba1857d75ae7cf9520ae0\": container with ID starting with 37ff1e4e649c408617b071e004fb1e8c7799cf02d8fba1857d75ae7cf9520ae0 not found: ID does not exist" containerID="37ff1e4e649c408617b071e004fb1e8c7799cf02d8fba1857d75ae7cf9520ae0"
May 20 15:54:33.505289 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.505229    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ff1e4e649c408617b071e004fb1e8c7799cf02d8fba1857d75ae7cf9520ae0"} err="failed to get container status \"37ff1e4e649c408617b071e004fb1e8c7799cf02d8fba1857d75ae7cf9520ae0\": rpc error: code = NotFound desc = could not find container \"37ff1e4e649c408617b071e004fb1e8c7799cf02d8fba1857d75ae7cf9520ae0\": container with ID starting with 37ff1e4e649c408617b071e004fb1e8c7799cf02d8fba1857d75ae7cf9520ae0 not found: ID does not exist"
May 20 15:54:33.505289 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.505248    2568 scope.go:117] "RemoveContainer" containerID="19dcbe5fb0f248c40b43620817e47f97fb47fd189e087afa1e8b252727df2f85"
May 20 15:54:33.505470 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:54:33.505457    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19dcbe5fb0f248c40b43620817e47f97fb47fd189e087afa1e8b252727df2f85\": container with ID starting with 19dcbe5fb0f248c40b43620817e47f97fb47fd189e087afa1e8b252727df2f85 not found: ID does not exist" containerID="19dcbe5fb0f248c40b43620817e47f97fb47fd189e087afa1e8b252727df2f85"
May 20 15:54:33.505510 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.505472    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19dcbe5fb0f248c40b43620817e47f97fb47fd189e087afa1e8b252727df2f85"} err="failed to get container status \"19dcbe5fb0f248c40b43620817e47f97fb47fd189e087afa1e8b252727df2f85\": rpc error: code = NotFound desc = could not find container \"19dcbe5fb0f248c40b43620817e47f97fb47fd189e087afa1e8b252727df2f85\": container with ID starting with 19dcbe5fb0f248c40b43620817e47f97fb47fd189e087afa1e8b252727df2f85 not found: ID does not exist"
May 20 15:54:33.508485 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.508460    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"]
May 20 15:54:33.512350 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.512319    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-23504-predictor-557c78fdfd-f6js7"]
May 20 15:54:33.645103 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.645057    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7922d8ef-19c4-4ef9-a09b-f16db6cf33e5" path="/var/lib/kubelet/pods/7922d8ef-19c4-4ef9-a09b-f16db6cf33e5/volumes"
May 20 15:54:33.738457 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.738433    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"
May 20 15:54:33.842009 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.841949    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77rd7\" (UniqueName: \"kubernetes.io/projected/45e8ec2e-cb37-46fd-9bca-db9bcd707e33-kube-api-access-77rd7\") pod \"45e8ec2e-cb37-46fd-9bca-db9bcd707e33\" (UID: \"45e8ec2e-cb37-46fd-9bca-db9bcd707e33\") "
May 20 15:54:33.842251 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.842051    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-23504-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45e8ec2e-cb37-46fd-9bca-db9bcd707e33-error-404-isvc-23504-kube-rbac-proxy-sar-config\") pod \"45e8ec2e-cb37-46fd-9bca-db9bcd707e33\" (UID: \"45e8ec2e-cb37-46fd-9bca-db9bcd707e33\") "
May 20 15:54:33.842251 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.842121    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45e8ec2e-cb37-46fd-9bca-db9bcd707e33-proxy-tls\") pod \"45e8ec2e-cb37-46fd-9bca-db9bcd707e33\" (UID: \"45e8ec2e-cb37-46fd-9bca-db9bcd707e33\") "
May 20 15:54:33.842418 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.842385    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e8ec2e-cb37-46fd-9bca-db9bcd707e33-error-404-isvc-23504-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-23504-kube-rbac-proxy-sar-config") pod "45e8ec2e-cb37-46fd-9bca-db9bcd707e33" (UID: "45e8ec2e-cb37-46fd-9bca-db9bcd707e33"). InnerVolumeSpecName "error-404-isvc-23504-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:54:33.844293 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.844271    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e8ec2e-cb37-46fd-9bca-db9bcd707e33-kube-api-access-77rd7" (OuterVolumeSpecName: "kube-api-access-77rd7") pod "45e8ec2e-cb37-46fd-9bca-db9bcd707e33" (UID: "45e8ec2e-cb37-46fd-9bca-db9bcd707e33"). InnerVolumeSpecName "kube-api-access-77rd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:54:33.844293 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.844283    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45e8ec2e-cb37-46fd-9bca-db9bcd707e33-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45e8ec2e-cb37-46fd-9bca-db9bcd707e33" (UID: "45e8ec2e-cb37-46fd-9bca-db9bcd707e33"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:54:33.943198 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.943104    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45e8ec2e-cb37-46fd-9bca-db9bcd707e33-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:54:33.943198 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.943136    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77rd7\" (UniqueName: \"kubernetes.io/projected/45e8ec2e-cb37-46fd-9bca-db9bcd707e33-kube-api-access-77rd7\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:54:33.943198 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:33.943149    2568 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-23504-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45e8ec2e-cb37-46fd-9bca-db9bcd707e33-error-404-isvc-23504-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:54:34.174109 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.174075    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c"
May 20 15:54:34.246915 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.246826    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ba7204-c40a-4fdc-97de-6257e21c8155-openshift-service-ca-bundle\") pod \"82ba7204-c40a-4fdc-97de-6257e21c8155\" (UID: \"82ba7204-c40a-4fdc-97de-6257e21c8155\") "
May 20 15:54:34.246915 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.246877    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82ba7204-c40a-4fdc-97de-6257e21c8155-proxy-tls\") pod \"82ba7204-c40a-4fdc-97de-6257e21c8155\" (UID: \"82ba7204-c40a-4fdc-97de-6257e21c8155\") "
May 20 15:54:34.247273 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.247249    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ba7204-c40a-4fdc-97de-6257e21c8155-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "82ba7204-c40a-4fdc-97de-6257e21c8155" (UID: "82ba7204-c40a-4fdc-97de-6257e21c8155"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:54:34.249190 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.249171    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82ba7204-c40a-4fdc-97de-6257e21c8155-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "82ba7204-c40a-4fdc-97de-6257e21c8155" (UID: "82ba7204-c40a-4fdc-97de-6257e21c8155"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:54:34.348422 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.348377    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ba7204-c40a-4fdc-97de-6257e21c8155-openshift-service-ca-bundle\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:54:34.348422 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.348415    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82ba7204-c40a-4fdc-97de-6257e21c8155-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:54:34.493861 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.493825    2568 generic.go:358] "Generic (PLEG): container finished" podID="82ba7204-c40a-4fdc-97de-6257e21c8155" containerID="6a9f9e19972e327a0634387ab9476f6bec7da6a0f853aa7f98b9f585b2621af4" exitCode=0
May 20 15:54:34.494331 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.493887    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c"
May 20 15:54:34.494331 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.493909    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c" event={"ID":"82ba7204-c40a-4fdc-97de-6257e21c8155","Type":"ContainerDied","Data":"6a9f9e19972e327a0634387ab9476f6bec7da6a0f853aa7f98b9f585b2621af4"}
May 20 15:54:34.494331 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.493939    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c" event={"ID":"82ba7204-c40a-4fdc-97de-6257e21c8155","Type":"ContainerDied","Data":"4b4605cca90cd72915080bc92ca82dfa2992286ef5a73af0f0a3336db6347b03"}
May 20 15:54:34.494331 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.493955    2568 scope.go:117] "RemoveContainer" containerID="6a9f9e19972e327a0634387ab9476f6bec7da6a0f853aa7f98b9f585b2621af4"
May 20 15:54:34.495445 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.495418    2568 generic.go:358] "Generic (PLEG): container finished" podID="45e8ec2e-cb37-46fd-9bca-db9bcd707e33" containerID="74c2486c69dab3b26ad1e6fbb2b88bfbeb1abdb9660e118e1fce209ec41451de" exitCode=0
May 20 15:54:34.495558 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.495462    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6" event={"ID":"45e8ec2e-cb37-46fd-9bca-db9bcd707e33","Type":"ContainerDied","Data":"74c2486c69dab3b26ad1e6fbb2b88bfbeb1abdb9660e118e1fce209ec41451de"}
May 20 15:54:34.495558 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.495483    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"
May 20 15:54:34.495558 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.495489    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6" event={"ID":"45e8ec2e-cb37-46fd-9bca-db9bcd707e33","Type":"ContainerDied","Data":"70181150ff39f2d998c34fd4a94c1d600e5152c716b666e5074494d81f9cf1ab"}
May 20 15:54:34.502755 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.502728    2568 scope.go:117] "RemoveContainer" containerID="6a9f9e19972e327a0634387ab9476f6bec7da6a0f853aa7f98b9f585b2621af4"
May 20 15:54:34.502995 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:54:34.502971    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a9f9e19972e327a0634387ab9476f6bec7da6a0f853aa7f98b9f585b2621af4\": container with ID starting with 6a9f9e19972e327a0634387ab9476f6bec7da6a0f853aa7f98b9f585b2621af4 not found: ID does not exist" containerID="6a9f9e19972e327a0634387ab9476f6bec7da6a0f853aa7f98b9f585b2621af4"
May 20 15:54:34.503118 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.503003    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a9f9e19972e327a0634387ab9476f6bec7da6a0f853aa7f98b9f585b2621af4"} err="failed to get container status \"6a9f9e19972e327a0634387ab9476f6bec7da6a0f853aa7f98b9f585b2621af4\": rpc error: code = NotFound desc = could not find container \"6a9f9e19972e327a0634387ab9476f6bec7da6a0f853aa7f98b9f585b2621af4\": container with ID starting with 6a9f9e19972e327a0634387ab9476f6bec7da6a0f853aa7f98b9f585b2621af4 not found: ID does not exist"
May 20 15:54:34.503118 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.503043    2568 scope.go:117] "RemoveContainer" containerID="ac3a54b558db5fcef62aab9c24890ce2233032a8ba2eca8f7aa73297fe90924c"
May 20 15:54:34.510783 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.510766    2568 scope.go:117] "RemoveContainer" containerID="74c2486c69dab3b26ad1e6fbb2b88bfbeb1abdb9660e118e1fce209ec41451de"
May 20 15:54:34.518939 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.518913    2568 scope.go:117] "RemoveContainer" containerID="ac3a54b558db5fcef62aab9c24890ce2233032a8ba2eca8f7aa73297fe90924c"
May 20 15:54:34.519248 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:54:34.519230    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3a54b558db5fcef62aab9c24890ce2233032a8ba2eca8f7aa73297fe90924c\": container with ID starting with ac3a54b558db5fcef62aab9c24890ce2233032a8ba2eca8f7aa73297fe90924c not found: ID does not exist" containerID="ac3a54b558db5fcef62aab9c24890ce2233032a8ba2eca8f7aa73297fe90924c"
May 20 15:54:34.519299 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.519259    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3a54b558db5fcef62aab9c24890ce2233032a8ba2eca8f7aa73297fe90924c"} err="failed to get container status \"ac3a54b558db5fcef62aab9c24890ce2233032a8ba2eca8f7aa73297fe90924c\": rpc error: code = NotFound desc = could not find container \"ac3a54b558db5fcef62aab9c24890ce2233032a8ba2eca8f7aa73297fe90924c\": container with ID starting with ac3a54b558db5fcef62aab9c24890ce2233032a8ba2eca8f7aa73297fe90924c not found: ID does not exist"
May 20 15:54:34.519299 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.519277    2568 scope.go:117] "RemoveContainer" containerID="74c2486c69dab3b26ad1e6fbb2b88bfbeb1abdb9660e118e1fce209ec41451de"
May 20 15:54:34.519516 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:54:34.519492    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74c2486c69dab3b26ad1e6fbb2b88bfbeb1abdb9660e118e1fce209ec41451de\": container with ID starting with 74c2486c69dab3b26ad1e6fbb2b88bfbeb1abdb9660e118e1fce209ec41451de not found: ID does not exist" containerID="74c2486c69dab3b26ad1e6fbb2b88bfbeb1abdb9660e118e1fce209ec41451de"
May 20 15:54:34.519601 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.519523    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74c2486c69dab3b26ad1e6fbb2b88bfbeb1abdb9660e118e1fce209ec41451de"} err="failed to get container status \"74c2486c69dab3b26ad1e6fbb2b88bfbeb1abdb9660e118e1fce209ec41451de\": rpc error: code = NotFound desc = could not find container \"74c2486c69dab3b26ad1e6fbb2b88bfbeb1abdb9660e118e1fce209ec41451de\": container with ID starting with 74c2486c69dab3b26ad1e6fbb2b88bfbeb1abdb9660e118e1fce209ec41451de not found: ID does not exist"
May 20 15:54:34.522462 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.522435    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"]
May 20 15:54:34.528426 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.528404    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-23504-predictor-6d476745d5-rhhv6"]
May 20 15:54:34.537667 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.537639    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c"]
May 20 15:54:34.542735 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:34.542712    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d4d66-58988c469-qj57c"]
May 20 15:54:35.643577 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:35.643540    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45e8ec2e-cb37-46fd-9bca-db9bcd707e33" path="/var/lib/kubelet/pods/45e8ec2e-cb37-46fd-9bca-db9bcd707e33/volumes"
May 20 15:54:35.643963 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:35.643931    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82ba7204-c40a-4fdc-97de-6257e21c8155" path="/var/lib/kubelet/pods/82ba7204-c40a-4fdc-97de-6257e21c8155/volumes"
May 20 15:54:36.699101 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:36.699060    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29" podUID="7d23dde2-c954-42ec-84b2-2686d2e578ad" containerName="sequence-graph-23504" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:54:37.488593 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:37.488561    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"
May 20 15:54:37.489111 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:37.489077    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp" podUID="da04dcf0-5363-4b02-b5ba-8e39acd77721" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
May 20 15:54:38.493144 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:38.493112    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"
May 20 15:54:38.493698 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:38.493666    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp" podUID="bbffac89-efb6-4741-b082-16c3d625f6b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
May 20 15:54:41.379263 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:41.379222    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45" podUID="41bf512d-9315-457b-a451-d7fb1eb230d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 20 15:54:41.379672 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:41.379352    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw" podUID="16b42b9f-fa74-4e57-8b69-f166541af5bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
May 20 15:54:41.698939 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:41.698847    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29" podUID="7d23dde2-c954-42ec-84b2-2686d2e578ad" containerName="sequence-graph-23504" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:54:41.699125 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:41.698971    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29"
May 20 15:54:46.698609 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:46.698567    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29" podUID="7d23dde2-c954-42ec-84b2-2686d2e578ad" containerName="sequence-graph-23504" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:54:47.489841 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:47.489799    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp" podUID="da04dcf0-5363-4b02-b5ba-8e39acd77721" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
May 20 15:54:48.493857 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:48.493812    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp" podUID="bbffac89-efb6-4741-b082-16c3d625f6b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
May 20 15:54:51.379364 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:51.379322    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45" podUID="41bf512d-9315-457b-a451-d7fb1eb230d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 20 15:54:51.379895 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:51.379874    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"
May 20 15:54:51.699320 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:51.699230    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29" podUID="7d23dde2-c954-42ec-84b2-2686d2e578ad" containerName="sequence-graph-23504" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:54:56.698588 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:56.698548    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29" podUID="7d23dde2-c954-42ec-84b2-2686d2e578ad" containerName="sequence-graph-23504" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:54:57.490150 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:57.490110    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp" podUID="da04dcf0-5363-4b02-b5ba-8e39acd77721" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
May 20 15:54:58.494280 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:58.494241    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp" podUID="bbffac89-efb6-4741-b082-16c3d625f6b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
May 20 15:54:59.461423 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:59.461396    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29"
May 20 15:54:59.568594 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:59.568493    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d23dde2-c954-42ec-84b2-2686d2e578ad-proxy-tls\") pod \"7d23dde2-c954-42ec-84b2-2686d2e578ad\" (UID: \"7d23dde2-c954-42ec-84b2-2686d2e578ad\") "
May 20 15:54:59.568594 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:59.568571    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d23dde2-c954-42ec-84b2-2686d2e578ad-openshift-service-ca-bundle\") pod \"7d23dde2-c954-42ec-84b2-2686d2e578ad\" (UID: \"7d23dde2-c954-42ec-84b2-2686d2e578ad\") "
May 20 15:54:59.569241 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:59.568935    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d23dde2-c954-42ec-84b2-2686d2e578ad-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "7d23dde2-c954-42ec-84b2-2686d2e578ad" (UID: "7d23dde2-c954-42ec-84b2-2686d2e578ad"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:54:59.570823 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:59.570794    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d23dde2-c954-42ec-84b2-2686d2e578ad-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7d23dde2-c954-42ec-84b2-2686d2e578ad" (UID: "7d23dde2-c954-42ec-84b2-2686d2e578ad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:54:59.595447 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:59.595410    2568 generic.go:358] "Generic (PLEG): container finished" podID="7d23dde2-c954-42ec-84b2-2686d2e578ad" containerID="4045ca7d46594da939e0517d0d54e65b5485819637f0daa4004094e9e1d164ef" exitCode=0
May 20 15:54:59.595578 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:59.595476    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29"
May 20 15:54:59.595578 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:59.595500    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29" event={"ID":"7d23dde2-c954-42ec-84b2-2686d2e578ad","Type":"ContainerDied","Data":"4045ca7d46594da939e0517d0d54e65b5485819637f0daa4004094e9e1d164ef"}
May 20 15:54:59.595578 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:59.595540    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29" event={"ID":"7d23dde2-c954-42ec-84b2-2686d2e578ad","Type":"ContainerDied","Data":"bbcc6fc476dceb3280d89f4c2d6f20fa96237fc93fe8227699f3612b8303f03d"}
May 20 15:54:59.595578 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:59.595555    2568 scope.go:117] "RemoveContainer" containerID="4045ca7d46594da939e0517d0d54e65b5485819637f0daa4004094e9e1d164ef"
May 20 15:54:59.604252 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:59.604235    2568 scope.go:117] "RemoveContainer" containerID="4045ca7d46594da939e0517d0d54e65b5485819637f0daa4004094e9e1d164ef"
May 20 15:54:59.604501 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:54:59.604483    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4045ca7d46594da939e0517d0d54e65b5485819637f0daa4004094e9e1d164ef\": container with ID starting with 4045ca7d46594da939e0517d0d54e65b5485819637f0daa4004094e9e1d164ef not found: ID does not exist" containerID="4045ca7d46594da939e0517d0d54e65b5485819637f0daa4004094e9e1d164ef"
May 20 15:54:59.604559 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:59.604510    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4045ca7d46594da939e0517d0d54e65b5485819637f0daa4004094e9e1d164ef"} err="failed to get container status \"4045ca7d46594da939e0517d0d54e65b5485819637f0daa4004094e9e1d164ef\": rpc error: code = NotFound desc = could not find container \"4045ca7d46594da939e0517d0d54e65b5485819637f0daa4004094e9e1d164ef\": container with ID starting with 4045ca7d46594da939e0517d0d54e65b5485819637f0daa4004094e9e1d164ef not found: ID does not exist"
May 20 15:54:59.616084 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:59.616057    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29"]
May 20 15:54:59.621277 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:59.621255    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-23504-54f4b4d8bc-pqd29"]
May 20 15:54:59.644108 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:59.644080    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d23dde2-c954-42ec-84b2-2686d2e578ad" path="/var/lib/kubelet/pods/7d23dde2-c954-42ec-84b2-2686d2e578ad/volumes"
May 20 15:54:59.669993 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:59.669969    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d23dde2-c954-42ec-84b2-2686d2e578ad-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:54:59.669993 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:54:59.669996    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d23dde2-c954-42ec-84b2-2686d2e578ad-openshift-service-ca-bundle\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:55:01.379949 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:01.379919    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"
May 20 15:55:07.489694 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:07.489654    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp" podUID="da04dcf0-5363-4b02-b5ba-8e39acd77721" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
May 20 15:55:08.494516 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:08.494474    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp" podUID="bbffac89-efb6-4741-b082-16c3d625f6b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
May 20 15:55:13.777370 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.777336    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c"]
May 20 15:55:13.777891 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.777863    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ba7204-c40a-4fdc-97de-6257e21c8155" containerName="switch-graph-d4d66"
May 20 15:55:13.777891 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.777885    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ba7204-c40a-4fdc-97de-6257e21c8155" containerName="switch-graph-d4d66"
May 20 15:55:13.778035 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.777904    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45e8ec2e-cb37-46fd-9bca-db9bcd707e33" containerName="kube-rbac-proxy"
May 20 15:55:13.778035 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.777910    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="45e8ec2e-cb37-46fd-9bca-db9bcd707e33" containerName="kube-rbac-proxy"
May 20 15:55:13.778035 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.777935    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d23dde2-c954-42ec-84b2-2686d2e578ad" containerName="sequence-graph-23504"
May 20 15:55:13.778035 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.777946    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d23dde2-c954-42ec-84b2-2686d2e578ad" containerName="sequence-graph-23504"
May 20 15:55:13.778035 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.777958    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7922d8ef-19c4-4ef9-a09b-f16db6cf33e5" containerName="kserve-container"
May 20 15:55:13.778035 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.777963    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7922d8ef-19c4-4ef9-a09b-f16db6cf33e5" containerName="kserve-container"
May 20 15:55:13.778035 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.777973    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45e8ec2e-cb37-46fd-9bca-db9bcd707e33" containerName="kserve-container"
May 20 15:55:13.778035 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.777979    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="45e8ec2e-cb37-46fd-9bca-db9bcd707e33" containerName="kserve-container"
May 20 15:55:13.778035 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.777990    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7922d8ef-19c4-4ef9-a09b-f16db6cf33e5" containerName="kube-rbac-proxy"
May 20 15:55:13.778035 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.777996    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7922d8ef-19c4-4ef9-a09b-f16db6cf33e5" containerName="kube-rbac-proxy"
May 20 15:55:13.778341 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.778079    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="45e8ec2e-cb37-46fd-9bca-db9bcd707e33" containerName="kserve-container"
May 20 15:55:13.778341 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.778089    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="82ba7204-c40a-4fdc-97de-6257e21c8155" containerName="switch-graph-d4d66"
May 20 15:55:13.778341 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.778097    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="45e8ec2e-cb37-46fd-9bca-db9bcd707e33" containerName="kube-rbac-proxy"
May 20 15:55:13.778341 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.778105    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="7d23dde2-c954-42ec-84b2-2686d2e578ad" containerName="sequence-graph-23504"
May 20 15:55:13.778341 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.778114    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="7922d8ef-19c4-4ef9-a09b-f16db6cf33e5" containerName="kube-rbac-proxy"
May 20 15:55:13.778341 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.778120    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="7922d8ef-19c4-4ef9-a09b-f16db6cf33e5" containerName="kserve-container"
May 20 15:55:13.782302 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.782285    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c"
May 20 15:55:13.785705 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.785685    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-687f8-serving-cert\""
May 20 15:55:13.785921 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.785900    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-687f8-kube-rbac-proxy-sar-config\""
May 20 15:55:13.791376 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.791347    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c"]
May 20 15:55:13.894479 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.894444    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13eb4007-0f04-4f0d-8768-7a3de2ab17e4-openshift-service-ca-bundle\") pod \"ensemble-graph-687f8-5b9ff78587-xmm2c\" (UID: \"13eb4007-0f04-4f0d-8768-7a3de2ab17e4\") " pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c"
May 20 15:55:13.894654 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.894492    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13eb4007-0f04-4f0d-8768-7a3de2ab17e4-proxy-tls\") pod \"ensemble-graph-687f8-5b9ff78587-xmm2c\" (UID: \"13eb4007-0f04-4f0d-8768-7a3de2ab17e4\") " pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c"
May 20 15:55:13.995128 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.995091    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13eb4007-0f04-4f0d-8768-7a3de2ab17e4-openshift-service-ca-bundle\") pod \"ensemble-graph-687f8-5b9ff78587-xmm2c\" (UID: \"13eb4007-0f04-4f0d-8768-7a3de2ab17e4\") " pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c"
May 20 15:55:13.995306 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.995139    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13eb4007-0f04-4f0d-8768-7a3de2ab17e4-proxy-tls\") pod \"ensemble-graph-687f8-5b9ff78587-xmm2c\" (UID: \"13eb4007-0f04-4f0d-8768-7a3de2ab17e4\") " pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c"
May 20 15:55:13.995359 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:55:13.995318    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-687f8-serving-cert: secret "ensemble-graph-687f8-serving-cert" not found
May 20 15:55:13.995402 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:55:13.995393    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13eb4007-0f04-4f0d-8768-7a3de2ab17e4-proxy-tls podName:13eb4007-0f04-4f0d-8768-7a3de2ab17e4 nodeName:}" failed. No retries permitted until 2026-05-20 15:55:14.495376303 +0000 UTC m=+1423.434217849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/13eb4007-0f04-4f0d-8768-7a3de2ab17e4-proxy-tls") pod "ensemble-graph-687f8-5b9ff78587-xmm2c" (UID: "13eb4007-0f04-4f0d-8768-7a3de2ab17e4") : secret "ensemble-graph-687f8-serving-cert" not found
May 20 15:55:13.995727 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:13.995707    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13eb4007-0f04-4f0d-8768-7a3de2ab17e4-openshift-service-ca-bundle\") pod \"ensemble-graph-687f8-5b9ff78587-xmm2c\" (UID: \"13eb4007-0f04-4f0d-8768-7a3de2ab17e4\") " pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c"
May 20 15:55:14.500683 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:14.500647    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13eb4007-0f04-4f0d-8768-7a3de2ab17e4-proxy-tls\") pod \"ensemble-graph-687f8-5b9ff78587-xmm2c\" (UID: \"13eb4007-0f04-4f0d-8768-7a3de2ab17e4\") " pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c"
May 20 15:55:14.503266 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:14.503235    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13eb4007-0f04-4f0d-8768-7a3de2ab17e4-proxy-tls\") pod \"ensemble-graph-687f8-5b9ff78587-xmm2c\" (UID: \"13eb4007-0f04-4f0d-8768-7a3de2ab17e4\") " pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c"
May 20 15:55:14.694230 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:14.694198    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c"
May 20 15:55:14.853344 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:14.853311    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c"]
May 20 15:55:14.856688 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:55:14.856651    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13eb4007_0f04_4f0d_8768_7a3de2ab17e4.slice/crio-8882501e059416abaefda6a3080f2aa9a5a52e153e7a38b26115eba5aa0b1a1e WatchSource:0}: Error finding container 8882501e059416abaefda6a3080f2aa9a5a52e153e7a38b26115eba5aa0b1a1e: Status 404 returned error can't find the container with id 8882501e059416abaefda6a3080f2aa9a5a52e153e7a38b26115eba5aa0b1a1e
May 20 15:55:15.652160 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:15.652121    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c" event={"ID":"13eb4007-0f04-4f0d-8768-7a3de2ab17e4","Type":"ContainerStarted","Data":"3fb38dec2151ae7ac659bf30c318c6f42d8832883b192681f5ec19a128a4da75"}
May 20 15:55:15.652160 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:15.652163    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c" event={"ID":"13eb4007-0f04-4f0d-8768-7a3de2ab17e4","Type":"ContainerStarted","Data":"8882501e059416abaefda6a3080f2aa9a5a52e153e7a38b26115eba5aa0b1a1e"}
May 20 15:55:15.652388 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:15.652255    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c"
May 20 15:55:15.676905 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:15.676847    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c" podStartSLOduration=2.6768279230000003 podStartE2EDuration="2.676827923s" podCreationTimestamp="2026-05-20 15:55:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:55:15.675524908 +0000 UTC m=+1424.614366464" watchObservedRunningTime="2026-05-20 15:55:15.676827923 +0000 UTC m=+1424.615669482"
May 20 15:55:17.489193 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:17.489155    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp" podUID="da04dcf0-5363-4b02-b5ba-8e39acd77721" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
May 20 15:55:18.495014 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:18.494987    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"
May 20 15:55:21.662333 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:21.662304    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c"
May 20 15:55:23.824496 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:23.824463    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c"]
May 20 15:55:23.824884 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:23.824773    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c" podUID="13eb4007-0f04-4f0d-8768-7a3de2ab17e4" containerName="ensemble-graph-687f8" containerID="cri-o://3fb38dec2151ae7ac659bf30c318c6f42d8832883b192681f5ec19a128a4da75" gracePeriod=30
May 20 15:55:23.935053 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:23.935002    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"]
May 20 15:55:23.935349 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:23.935319    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45" podUID="41bf512d-9315-457b-a451-d7fb1eb230d4" containerName="kserve-container" containerID="cri-o://8180acaa5a3968e0e3528da7b66576021e423ee45536140450de254427187960" gracePeriod=30
May 20 15:55:23.935564 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:23.935502    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45" podUID="41bf512d-9315-457b-a451-d7fb1eb230d4" containerName="kube-rbac-proxy" containerID="cri-o://81ceaa5ac9b87b883e20fca58d954bdac96e9821af2755ab576bff7a0a476fef" gracePeriod=30
May 20 15:55:23.973513 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:23.973478    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"]
May 20 15:55:23.977416 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:23.977392    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"
May 20 15:55:23.979759 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:23.979723    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3f928-predictor-serving-cert\""
May 20 15:55:23.979759 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:23.979747    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3f928-kube-rbac-proxy-sar-config\""
May 20 15:55:23.990102 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:23.988911    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"]
May 20 15:55:24.024323 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.024286    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"]
May 20 15:55:24.024761 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.024702    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw" podUID="16b42b9f-fa74-4e57-8b69-f166541af5bd" containerName="kserve-container" containerID="cri-o://10b26717daa00dfc2c0f68332cd1723c95fadb40d133e855595eb44d1d43577d" gracePeriod=30
May 20 15:55:24.025011 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.024789    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw" podUID="16b42b9f-fa74-4e57-8b69-f166541af5bd" containerName="kube-rbac-proxy" containerID="cri-o://f89266cb60c15bb10e64d197cd7b8fdad39b4cab9c4db78c8a2e4e279d20e097" gracePeriod=30
May 20 15:55:24.080541 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.080448    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"]
May 20 15:55:24.084696 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.084676    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"
May 20 15:55:24.087167 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.087146    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3f928-kube-rbac-proxy-sar-config\""
May 20 15:55:24.087269 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.087146    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3f928-predictor-serving-cert\""
May 20 15:55:24.090000 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.089966    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbrc2\" (UniqueName: \"kubernetes.io/projected/022a7db3-b507-4127-9278-3d3308028d02-kube-api-access-vbrc2\") pod \"success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c\" (UID: \"022a7db3-b507-4127-9278-3d3308028d02\") " pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"
May 20 15:55:24.090143 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.090062    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-3f928-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/022a7db3-b507-4127-9278-3d3308028d02-success-200-isvc-3f928-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c\" (UID: \"022a7db3-b507-4127-9278-3d3308028d02\") " pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"
May 20 15:55:24.090143 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.090120    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/022a7db3-b507-4127-9278-3d3308028d02-proxy-tls\") pod \"success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c\" (UID: \"022a7db3-b507-4127-9278-3d3308028d02\") " pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"
May 20 15:55:24.091286 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.091264    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"]
May 20 15:55:24.191404 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.191363    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vbrc2\" (UniqueName: \"kubernetes.io/projected/022a7db3-b507-4127-9278-3d3308028d02-kube-api-access-vbrc2\") pod \"success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c\" (UID: \"022a7db3-b507-4127-9278-3d3308028d02\") " pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"
May 20 15:55:24.191605 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.191413    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-3f928-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b9ffefe-0aad-4ca0-9b52-59b073671f33-error-404-isvc-3f928-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3f928-predictor-787568c877-rvqzn\" (UID: \"0b9ffefe-0aad-4ca0-9b52-59b073671f33\") " pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"
May 20 15:55:24.191605 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.191452    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b9ffefe-0aad-4ca0-9b52-59b073671f33-proxy-tls\") pod \"error-404-isvc-3f928-predictor-787568c877-rvqzn\" (UID: \"0b9ffefe-0aad-4ca0-9b52-59b073671f33\") " pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"
May 20 15:55:24.191605 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.191484    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-3f928-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/022a7db3-b507-4127-9278-3d3308028d02-success-200-isvc-3f928-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c\" (UID: \"022a7db3-b507-4127-9278-3d3308028d02\") " pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"
May 20 15:55:24.191781 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.191605    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/022a7db3-b507-4127-9278-3d3308028d02-proxy-tls\") pod \"success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c\" (UID: \"022a7db3-b507-4127-9278-3d3308028d02\") " pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"
May 20 15:55:24.191781 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.191747    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59kdg\" (UniqueName: \"kubernetes.io/projected/0b9ffefe-0aad-4ca0-9b52-59b073671f33-kube-api-access-59kdg\") pod \"error-404-isvc-3f928-predictor-787568c877-rvqzn\" (UID: \"0b9ffefe-0aad-4ca0-9b52-59b073671f33\") " pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"
May 20 15:55:24.191887 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:55:24.191777    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-3f928-predictor-serving-cert: secret "success-200-isvc-3f928-predictor-serving-cert" not found
May 20 15:55:24.191887 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:55:24.191838    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/022a7db3-b507-4127-9278-3d3308028d02-proxy-tls podName:022a7db3-b507-4127-9278-3d3308028d02 nodeName:}" failed. No retries permitted until 2026-05-20 15:55:24.691817695 +0000 UTC m=+1433.630659233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/022a7db3-b507-4127-9278-3d3308028d02-proxy-tls") pod "success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c" (UID: "022a7db3-b507-4127-9278-3d3308028d02") : secret "success-200-isvc-3f928-predictor-serving-cert" not found
May 20 15:55:24.192277 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.192253    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-3f928-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/022a7db3-b507-4127-9278-3d3308028d02-success-200-isvc-3f928-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c\" (UID: \"022a7db3-b507-4127-9278-3d3308028d02\") " pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"
May 20 15:55:24.209663 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.209634    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbrc2\" (UniqueName: \"kubernetes.io/projected/022a7db3-b507-4127-9278-3d3308028d02-kube-api-access-vbrc2\") pod \"success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c\" (UID: \"022a7db3-b507-4127-9278-3d3308028d02\") " pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"
May 20 15:55:24.292857 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.292811    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-3f928-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b9ffefe-0aad-4ca0-9b52-59b073671f33-error-404-isvc-3f928-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3f928-predictor-787568c877-rvqzn\" (UID: \"0b9ffefe-0aad-4ca0-9b52-59b073671f33\") " pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"
May 20 15:55:24.293100 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.292863    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b9ffefe-0aad-4ca0-9b52-59b073671f33-proxy-tls\") pod \"error-404-isvc-3f928-predictor-787568c877-rvqzn\" (UID: \"0b9ffefe-0aad-4ca0-9b52-59b073671f33\") " pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"
May 20 15:55:24.293100 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.293006    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-59kdg\" (UniqueName: \"kubernetes.io/projected/0b9ffefe-0aad-4ca0-9b52-59b073671f33-kube-api-access-59kdg\") pod \"error-404-isvc-3f928-predictor-787568c877-rvqzn\" (UID: \"0b9ffefe-0aad-4ca0-9b52-59b073671f33\") " pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"
May 20 15:55:24.293100 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:55:24.293064    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-3f928-predictor-serving-cert: secret "error-404-isvc-3f928-predictor-serving-cert" not found
May 20 15:55:24.293277 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:55:24.293185    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b9ffefe-0aad-4ca0-9b52-59b073671f33-proxy-tls podName:0b9ffefe-0aad-4ca0-9b52-59b073671f33 nodeName:}" failed. No retries permitted until 2026-05-20 15:55:24.793162236 +0000 UTC m=+1433.732003794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0b9ffefe-0aad-4ca0-9b52-59b073671f33-proxy-tls") pod "error-404-isvc-3f928-predictor-787568c877-rvqzn" (UID: "0b9ffefe-0aad-4ca0-9b52-59b073671f33") : secret "error-404-isvc-3f928-predictor-serving-cert" not found
May 20 15:55:24.293602 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.293580    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-3f928-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b9ffefe-0aad-4ca0-9b52-59b073671f33-error-404-isvc-3f928-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3f928-predictor-787568c877-rvqzn\" (UID: \"0b9ffefe-0aad-4ca0-9b52-59b073671f33\") " pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"
May 20 15:55:24.313355 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.313317    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59kdg\" (UniqueName: \"kubernetes.io/projected/0b9ffefe-0aad-4ca0-9b52-59b073671f33-kube-api-access-59kdg\") pod \"error-404-isvc-3f928-predictor-787568c877-rvqzn\" (UID: \"0b9ffefe-0aad-4ca0-9b52-59b073671f33\") " pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"
May 20 15:55:24.683242 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.683150    2568 generic.go:358] "Generic (PLEG): container finished" podID="16b42b9f-fa74-4e57-8b69-f166541af5bd" containerID="f89266cb60c15bb10e64d197cd7b8fdad39b4cab9c4db78c8a2e4e279d20e097" exitCode=2
May 20 15:55:24.683242 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.683231    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw" event={"ID":"16b42b9f-fa74-4e57-8b69-f166541af5bd","Type":"ContainerDied","Data":"f89266cb60c15bb10e64d197cd7b8fdad39b4cab9c4db78c8a2e4e279d20e097"}
May 20 15:55:24.684814 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.684791    2568 generic.go:358] "Generic (PLEG): container finished" podID="41bf512d-9315-457b-a451-d7fb1eb230d4" containerID="81ceaa5ac9b87b883e20fca58d954bdac96e9821af2755ab576bff7a0a476fef" exitCode=2
May 20 15:55:24.684930 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.684857    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45" event={"ID":"41bf512d-9315-457b-a451-d7fb1eb230d4","Type":"ContainerDied","Data":"81ceaa5ac9b87b883e20fca58d954bdac96e9821af2755ab576bff7a0a476fef"}
May 20 15:55:24.696464 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.696413    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/022a7db3-b507-4127-9278-3d3308028d02-proxy-tls\") pod \"success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c\" (UID: \"022a7db3-b507-4127-9278-3d3308028d02\") " pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"
May 20 15:55:24.699058 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.699037    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/022a7db3-b507-4127-9278-3d3308028d02-proxy-tls\") pod \"success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c\" (UID: \"022a7db3-b507-4127-9278-3d3308028d02\") " pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"
May 20 15:55:24.797470 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.797414    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b9ffefe-0aad-4ca0-9b52-59b073671f33-proxy-tls\") pod \"error-404-isvc-3f928-predictor-787568c877-rvqzn\" (UID: \"0b9ffefe-0aad-4ca0-9b52-59b073671f33\") " pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"
May 20 15:55:24.800102 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.800069    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b9ffefe-0aad-4ca0-9b52-59b073671f33-proxy-tls\") pod \"error-404-isvc-3f928-predictor-787568c877-rvqzn\" (UID: \"0b9ffefe-0aad-4ca0-9b52-59b073671f33\") " pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"
May 20 15:55:24.894792 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.894744    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"
May 20 15:55:24.998424 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:24.998368    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"
May 20 15:55:25.065607 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:25.062155    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"]
May 20 15:55:25.066395 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:55:25.066360    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022a7db3_b507_4127_9278_3d3308028d02.slice/crio-9a092d017ea03dc37d676bb2e480c142d8225520b99ae88c648ca7093eb423ad WatchSource:0}: Error finding container 9a092d017ea03dc37d676bb2e480c142d8225520b99ae88c648ca7093eb423ad: Status 404 returned error can't find the container with id 9a092d017ea03dc37d676bb2e480c142d8225520b99ae88c648ca7093eb423ad
May 20 15:55:25.141956 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:25.141820    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"]
May 20 15:55:25.144047 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:55:25.144002    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9ffefe_0aad_4ca0_9b52_59b073671f33.slice/crio-59798dc8cb3ef9d84c64f2c446073b398b6a713a7f99505e9a087c71170fe381 WatchSource:0}: Error finding container 59798dc8cb3ef9d84c64f2c446073b398b6a713a7f99505e9a087c71170fe381: Status 404 returned error can't find the container with id 59798dc8cb3ef9d84c64f2c446073b398b6a713a7f99505e9a087c71170fe381
May 20 15:55:25.689642 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:25.689601    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn" event={"ID":"0b9ffefe-0aad-4ca0-9b52-59b073671f33","Type":"ContainerStarted","Data":"ea1db858048e39fe0a046621f042a4b019c06a79ced6b768e81332e2f3bb050c"}
May 20 15:55:25.689642 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:25.689643    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn" event={"ID":"0b9ffefe-0aad-4ca0-9b52-59b073671f33","Type":"ContainerStarted","Data":"c19f9a06f96c019063883ef931c4c936200894e7f6d5fc4ceb3a373ad451095a"}
May 20 15:55:25.689900 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:25.689657    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn" event={"ID":"0b9ffefe-0aad-4ca0-9b52-59b073671f33","Type":"ContainerStarted","Data":"59798dc8cb3ef9d84c64f2c446073b398b6a713a7f99505e9a087c71170fe381"}
May 20 15:55:25.689900 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:25.689720    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"
May 20 15:55:25.691228 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:25.691199    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c" event={"ID":"022a7db3-b507-4127-9278-3d3308028d02","Type":"ContainerStarted","Data":"3fbf0499141b51b71bff11f9f5b42c48b6001a3b10e2567e66f15e00c84d1471"}
May 20 15:55:25.691338 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:25.691235    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c" event={"ID":"022a7db3-b507-4127-9278-3d3308028d02","Type":"ContainerStarted","Data":"5f46792aecb8d4b5d9d2e6fe18dfca4d13ce32163a1e1722cb2ac47f87935801"}
May 20 15:55:25.691338 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:25.691250    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c" event={"ID":"022a7db3-b507-4127-9278-3d3308028d02","Type":"ContainerStarted","Data":"9a092d017ea03dc37d676bb2e480c142d8225520b99ae88c648ca7093eb423ad"}
May 20 15:55:25.691338 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:25.691270    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"
May 20 15:55:25.691338 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:25.691284    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"
May 20 15:55:25.692824 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:25.692794    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c" podUID="022a7db3-b507-4127-9278-3d3308028d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 20 15:55:25.708443 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:25.708399    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn" podStartSLOduration=1.708386214 podStartE2EDuration="1.708386214s" podCreationTimestamp="2026-05-20 15:55:24 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:55:25.706261654 +0000 UTC m=+1434.645103212" watchObservedRunningTime="2026-05-20 15:55:25.708386214 +0000 UTC m=+1434.647227770"
May 20 15:55:25.725666 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:25.725614    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c" podStartSLOduration=2.72559854 podStartE2EDuration="2.72559854s" podCreationTimestamp="2026-05-20 15:55:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:55:25.724515894 +0000 UTC m=+1434.663357450" watchObservedRunningTime="2026-05-20 15:55:25.72559854 +0000 UTC m=+1434.664440098"
May 20 15:55:26.373486 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:26.373432    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45" podUID="41bf512d-9315-457b-a451-d7fb1eb230d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
May 20 15:55:26.373996 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:26.373441    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw" podUID="16b42b9f-fa74-4e57-8b69-f166541af5bd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
May 20 15:55:26.661298 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:26.661199    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c" podUID="13eb4007-0f04-4f0d-8768-7a3de2ab17e4" containerName="ensemble-graph-687f8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:55:26.694938 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:26.694898    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"
May 20 15:55:26.695169 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:26.695104    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c" podUID="022a7db3-b507-4127-9278-3d3308028d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 20 15:55:26.696276 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:26.696249    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn" podUID="0b9ffefe-0aad-4ca0-9b52-59b073671f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
May 20 15:55:27.490199 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:27.490167    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"
May 20 15:55:27.698302 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:27.698261    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn" podUID="0b9ffefe-0aad-4ca0-9b52-59b073671f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
May 20 15:55:28.019299 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.019274    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"
May 20 15:55:28.022658 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.022635    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"
May 20 15:55:28.129305 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.129270    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6vwk\" (UniqueName: \"kubernetes.io/projected/16b42b9f-fa74-4e57-8b69-f166541af5bd-kube-api-access-w6vwk\") pod \"16b42b9f-fa74-4e57-8b69-f166541af5bd\" (UID: \"16b42b9f-fa74-4e57-8b69-f166541af5bd\") "
May 20 15:55:28.129504 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.129339    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16b42b9f-fa74-4e57-8b69-f166541af5bd-proxy-tls\") pod \"16b42b9f-fa74-4e57-8b69-f166541af5bd\" (UID: \"16b42b9f-fa74-4e57-8b69-f166541af5bd\") "
May 20 15:55:28.129504 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.129376    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l96ht\" (UniqueName: \"kubernetes.io/projected/41bf512d-9315-457b-a451-d7fb1eb230d4-kube-api-access-l96ht\") pod \"41bf512d-9315-457b-a451-d7fb1eb230d4\" (UID: \"41bf512d-9315-457b-a451-d7fb1eb230d4\") "
May 20 15:55:28.129504 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.129399    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-687f8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16b42b9f-fa74-4e57-8b69-f166541af5bd-error-404-isvc-687f8-kube-rbac-proxy-sar-config\") pod \"16b42b9f-fa74-4e57-8b69-f166541af5bd\" (UID: \"16b42b9f-fa74-4e57-8b69-f166541af5bd\") "
May 20 15:55:28.129504 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.129435    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41bf512d-9315-457b-a451-d7fb1eb230d4-proxy-tls\") pod \"41bf512d-9315-457b-a451-d7fb1eb230d4\" (UID: \"41bf512d-9315-457b-a451-d7fb1eb230d4\") "
May 20 15:55:28.129725 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.129554    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-687f8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41bf512d-9315-457b-a451-d7fb1eb230d4-success-200-isvc-687f8-kube-rbac-proxy-sar-config\") pod \"41bf512d-9315-457b-a451-d7fb1eb230d4\" (UID: \"41bf512d-9315-457b-a451-d7fb1eb230d4\") "
May 20 15:55:28.129879 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.129832    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b42b9f-fa74-4e57-8b69-f166541af5bd-error-404-isvc-687f8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-687f8-kube-rbac-proxy-sar-config") pod "16b42b9f-fa74-4e57-8b69-f166541af5bd" (UID: "16b42b9f-fa74-4e57-8b69-f166541af5bd"). InnerVolumeSpecName "error-404-isvc-687f8-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:55:28.129983 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.129953    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41bf512d-9315-457b-a451-d7fb1eb230d4-success-200-isvc-687f8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-687f8-kube-rbac-proxy-sar-config") pod "41bf512d-9315-457b-a451-d7fb1eb230d4" (UID: "41bf512d-9315-457b-a451-d7fb1eb230d4"). InnerVolumeSpecName "success-200-isvc-687f8-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:55:28.131627 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.131601    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41bf512d-9315-457b-a451-d7fb1eb230d4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "41bf512d-9315-457b-a451-d7fb1eb230d4" (UID: "41bf512d-9315-457b-a451-d7fb1eb230d4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:55:28.131743 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.131720    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41bf512d-9315-457b-a451-d7fb1eb230d4-kube-api-access-l96ht" (OuterVolumeSpecName: "kube-api-access-l96ht") pod "41bf512d-9315-457b-a451-d7fb1eb230d4" (UID: "41bf512d-9315-457b-a451-d7fb1eb230d4"). InnerVolumeSpecName "kube-api-access-l96ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:55:28.132036 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.132001    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16b42b9f-fa74-4e57-8b69-f166541af5bd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "16b42b9f-fa74-4e57-8b69-f166541af5bd" (UID: "16b42b9f-fa74-4e57-8b69-f166541af5bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:55:28.132094 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.132035    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b42b9f-fa74-4e57-8b69-f166541af5bd-kube-api-access-w6vwk" (OuterVolumeSpecName: "kube-api-access-w6vwk") pod "16b42b9f-fa74-4e57-8b69-f166541af5bd" (UID: "16b42b9f-fa74-4e57-8b69-f166541af5bd"). InnerVolumeSpecName "kube-api-access-w6vwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:55:28.230594 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.230547    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6vwk\" (UniqueName: \"kubernetes.io/projected/16b42b9f-fa74-4e57-8b69-f166541af5bd-kube-api-access-w6vwk\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:55:28.230594 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.230590    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16b42b9f-fa74-4e57-8b69-f166541af5bd-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:55:28.230594 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.230603    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l96ht\" (UniqueName: \"kubernetes.io/projected/41bf512d-9315-457b-a451-d7fb1eb230d4-kube-api-access-l96ht\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:55:28.230851 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.230615    2568 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-687f8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16b42b9f-fa74-4e57-8b69-f166541af5bd-error-404-isvc-687f8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:55:28.230851 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.230626    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41bf512d-9315-457b-a451-d7fb1eb230d4-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:55:28.230851 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.230636    2568 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-687f8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41bf512d-9315-457b-a451-d7fb1eb230d4-success-200-isvc-687f8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:55:28.703281 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.703244    2568 generic.go:358] "Generic (PLEG): container finished" podID="16b42b9f-fa74-4e57-8b69-f166541af5bd" containerID="10b26717daa00dfc2c0f68332cd1723c95fadb40d133e855595eb44d1d43577d" exitCode=0
May 20 15:55:28.703723 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.703318    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"
May 20 15:55:28.703723 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.703321    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw" event={"ID":"16b42b9f-fa74-4e57-8b69-f166541af5bd","Type":"ContainerDied","Data":"10b26717daa00dfc2c0f68332cd1723c95fadb40d133e855595eb44d1d43577d"}
May 20 15:55:28.703723 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.703352    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw" event={"ID":"16b42b9f-fa74-4e57-8b69-f166541af5bd","Type":"ContainerDied","Data":"05ee8a1087adfeb849cb78b75222177c51206bdc2da1f4c828499a9c81d42c95"}
May 20 15:55:28.703723 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.703372    2568 scope.go:117] "RemoveContainer" containerID="f89266cb60c15bb10e64d197cd7b8fdad39b4cab9c4db78c8a2e4e279d20e097"
May 20 15:55:28.704772 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.704741    2568 generic.go:358] "Generic (PLEG): container finished" podID="41bf512d-9315-457b-a451-d7fb1eb230d4" containerID="8180acaa5a3968e0e3528da7b66576021e423ee45536140450de254427187960" exitCode=0
May 20 15:55:28.704859 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.704808    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45" event={"ID":"41bf512d-9315-457b-a451-d7fb1eb230d4","Type":"ContainerDied","Data":"8180acaa5a3968e0e3528da7b66576021e423ee45536140450de254427187960"}
May 20 15:55:28.704859 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.704849    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45" event={"ID":"41bf512d-9315-457b-a451-d7fb1eb230d4","Type":"ContainerDied","Data":"466fb5601ba7301b6dec9ab9352bcf5b429a76f60dd989fc75c4d063d683887f"}
May 20 15:55:28.704974 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.704816    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"
May 20 15:55:28.712037 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.711997    2568 scope.go:117] "RemoveContainer" containerID="10b26717daa00dfc2c0f68332cd1723c95fadb40d133e855595eb44d1d43577d"
May 20 15:55:28.720537 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.720520    2568 scope.go:117] "RemoveContainer" containerID="f89266cb60c15bb10e64d197cd7b8fdad39b4cab9c4db78c8a2e4e279d20e097"
May 20 15:55:28.720806 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:55:28.720785    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f89266cb60c15bb10e64d197cd7b8fdad39b4cab9c4db78c8a2e4e279d20e097\": container with ID starting with f89266cb60c15bb10e64d197cd7b8fdad39b4cab9c4db78c8a2e4e279d20e097 not found: ID does not exist" containerID="f89266cb60c15bb10e64d197cd7b8fdad39b4cab9c4db78c8a2e4e279d20e097"
May 20 15:55:28.720857 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.720816    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89266cb60c15bb10e64d197cd7b8fdad39b4cab9c4db78c8a2e4e279d20e097"} err="failed to get container status \"f89266cb60c15bb10e64d197cd7b8fdad39b4cab9c4db78c8a2e4e279d20e097\": rpc error: code = NotFound desc = could not find container \"f89266cb60c15bb10e64d197cd7b8fdad39b4cab9c4db78c8a2e4e279d20e097\": container with ID starting with f89266cb60c15bb10e64d197cd7b8fdad39b4cab9c4db78c8a2e4e279d20e097 not found: ID does not exist"
May 20 15:55:28.720857 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.720834    2568 scope.go:117] "RemoveContainer" containerID="10b26717daa00dfc2c0f68332cd1723c95fadb40d133e855595eb44d1d43577d"
May 20 15:55:28.721123 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:55:28.721104    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10b26717daa00dfc2c0f68332cd1723c95fadb40d133e855595eb44d1d43577d\": container with ID starting with 10b26717daa00dfc2c0f68332cd1723c95fadb40d133e855595eb44d1d43577d not found: ID does not exist" containerID="10b26717daa00dfc2c0f68332cd1723c95fadb40d133e855595eb44d1d43577d"
May 20 15:55:28.721182 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.721128    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b26717daa00dfc2c0f68332cd1723c95fadb40d133e855595eb44d1d43577d"} err="failed to get container status \"10b26717daa00dfc2c0f68332cd1723c95fadb40d133e855595eb44d1d43577d\": rpc error: code = NotFound desc = could not find container \"10b26717daa00dfc2c0f68332cd1723c95fadb40d133e855595eb44d1d43577d\": container with ID starting with 10b26717daa00dfc2c0f68332cd1723c95fadb40d133e855595eb44d1d43577d not found: ID does not exist"
May 20 15:55:28.721182 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.721145    2568 scope.go:117] "RemoveContainer" containerID="81ceaa5ac9b87b883e20fca58d954bdac96e9821af2755ab576bff7a0a476fef"
May 20 15:55:28.729571 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.729544    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"]
May 20 15:55:28.730479 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.730464    2568 scope.go:117] "RemoveContainer" containerID="8180acaa5a3968e0e3528da7b66576021e423ee45536140450de254427187960"
May 20 15:55:28.731726 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.731710    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-687f8-predictor-7d75c44d4d-64c45"]
May 20 15:55:28.739327 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.739306    2568 scope.go:117] "RemoveContainer" containerID="81ceaa5ac9b87b883e20fca58d954bdac96e9821af2755ab576bff7a0a476fef"
May 20 15:55:28.739620 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:55:28.739601    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ceaa5ac9b87b883e20fca58d954bdac96e9821af2755ab576bff7a0a476fef\": container with ID starting with 81ceaa5ac9b87b883e20fca58d954bdac96e9821af2755ab576bff7a0a476fef not found: ID does not exist" containerID="81ceaa5ac9b87b883e20fca58d954bdac96e9821af2755ab576bff7a0a476fef"
May 20 15:55:28.739696 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.739636    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ceaa5ac9b87b883e20fca58d954bdac96e9821af2755ab576bff7a0a476fef"} err="failed to get container status \"81ceaa5ac9b87b883e20fca58d954bdac96e9821af2755ab576bff7a0a476fef\": rpc error: code = NotFound desc = could not find container \"81ceaa5ac9b87b883e20fca58d954bdac96e9821af2755ab576bff7a0a476fef\": container with ID starting with 81ceaa5ac9b87b883e20fca58d954bdac96e9821af2755ab576bff7a0a476fef not found: ID does not exist"
May 20 15:55:28.739696 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.739656    2568 scope.go:117] "RemoveContainer" containerID="8180acaa5a3968e0e3528da7b66576021e423ee45536140450de254427187960"
May 20 15:55:28.739882 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:55:28.739861    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8180acaa5a3968e0e3528da7b66576021e423ee45536140450de254427187960\": container with ID starting with 8180acaa5a3968e0e3528da7b66576021e423ee45536140450de254427187960 not found: ID does not exist" containerID="8180acaa5a3968e0e3528da7b66576021e423ee45536140450de254427187960"
May 20 15:55:28.739981 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.739886    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8180acaa5a3968e0e3528da7b66576021e423ee45536140450de254427187960"} err="failed to get container status \"8180acaa5a3968e0e3528da7b66576021e423ee45536140450de254427187960\": rpc error: code = NotFound desc = could not find container \"8180acaa5a3968e0e3528da7b66576021e423ee45536140450de254427187960\": container with ID starting with 8180acaa5a3968e0e3528da7b66576021e423ee45536140450de254427187960 not found: ID does not exist"
May 20 15:55:28.740623 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.740605    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"]
May 20 15:55:28.746782 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:28.746760    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-687f8-predictor-d97cfdf7f-2xmtw"]
May 20 15:55:29.487139 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.487107    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz"]
May 20 15:55:29.487504 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.487493    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41bf512d-9315-457b-a451-d7fb1eb230d4" containerName="kserve-container"
May 20 15:55:29.487545 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.487506    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="41bf512d-9315-457b-a451-d7fb1eb230d4" containerName="kserve-container"
May 20 15:55:29.487545 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.487527    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16b42b9f-fa74-4e57-8b69-f166541af5bd" containerName="kserve-container"
May 20 15:55:29.487545 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.487532    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="16b42b9f-fa74-4e57-8b69-f166541af5bd" containerName="kserve-container"
May 20 15:55:29.487545 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.487539    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41bf512d-9315-457b-a451-d7fb1eb230d4" containerName="kube-rbac-proxy"
May 20 15:55:29.487545 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.487544    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="41bf512d-9315-457b-a451-d7fb1eb230d4" containerName="kube-rbac-proxy"
May 20 15:55:29.487726 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.487552    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16b42b9f-fa74-4e57-8b69-f166541af5bd" containerName="kube-rbac-proxy"
May 20 15:55:29.487726 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.487557    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="16b42b9f-fa74-4e57-8b69-f166541af5bd" containerName="kube-rbac-proxy"
May 20 15:55:29.487726 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.487607    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="41bf512d-9315-457b-a451-d7fb1eb230d4" containerName="kserve-container"
May 20 15:55:29.487726 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.487616    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="16b42b9f-fa74-4e57-8b69-f166541af5bd" containerName="kserve-container"
May 20 15:55:29.487726 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.487625    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="16b42b9f-fa74-4e57-8b69-f166541af5bd" containerName="kube-rbac-proxy"
May 20 15:55:29.487726 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.487633    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="41bf512d-9315-457b-a451-d7fb1eb230d4" containerName="kube-rbac-proxy"
May 20 15:55:29.492528 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.492503    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz"
May 20 15:55:29.494700 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.494677    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-4ff76-kube-rbac-proxy-sar-config\""
May 20 15:55:29.494837 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.494682    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-4ff76-serving-cert\""
May 20 15:55:29.500827 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.500802    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz"]
May 20 15:55:29.644906 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.644872    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/763038c5-1d94-4e7e-88bb-a730362c7469-proxy-tls\") pod \"sequence-graph-4ff76-59c94c4c8c-4gzqz\" (UID: \"763038c5-1d94-4e7e-88bb-a730362c7469\") " pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz"
May 20 15:55:29.645113 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.644957    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/763038c5-1d94-4e7e-88bb-a730362c7469-openshift-service-ca-bundle\") pod \"sequence-graph-4ff76-59c94c4c8c-4gzqz\" (UID: \"763038c5-1d94-4e7e-88bb-a730362c7469\") " pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz"
May 20 15:55:29.645527 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.645507    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16b42b9f-fa74-4e57-8b69-f166541af5bd" path="/var/lib/kubelet/pods/16b42b9f-fa74-4e57-8b69-f166541af5bd/volumes"
May 20 15:55:29.645929 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.645914    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41bf512d-9315-457b-a451-d7fb1eb230d4" path="/var/lib/kubelet/pods/41bf512d-9315-457b-a451-d7fb1eb230d4/volumes"
May 20 15:55:29.745779 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.745689    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/763038c5-1d94-4e7e-88bb-a730362c7469-proxy-tls\") pod \"sequence-graph-4ff76-59c94c4c8c-4gzqz\" (UID: \"763038c5-1d94-4e7e-88bb-a730362c7469\") " pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz"
May 20 15:55:29.745779 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.745731    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/763038c5-1d94-4e7e-88bb-a730362c7469-openshift-service-ca-bundle\") pod \"sequence-graph-4ff76-59c94c4c8c-4gzqz\" (UID: \"763038c5-1d94-4e7e-88bb-a730362c7469\") " pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz"
May 20 15:55:29.746239 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:55:29.745843    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-4ff76-serving-cert: secret "sequence-graph-4ff76-serving-cert" not found
May 20 15:55:29.746239 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:55:29.745908    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/763038c5-1d94-4e7e-88bb-a730362c7469-proxy-tls podName:763038c5-1d94-4e7e-88bb-a730362c7469 nodeName:}" failed. No retries permitted until 2026-05-20 15:55:30.245891308 +0000 UTC m=+1439.184732843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/763038c5-1d94-4e7e-88bb-a730362c7469-proxy-tls") pod "sequence-graph-4ff76-59c94c4c8c-4gzqz" (UID: "763038c5-1d94-4e7e-88bb-a730362c7469") : secret "sequence-graph-4ff76-serving-cert" not found
May 20 15:55:29.746356 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:29.746338    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/763038c5-1d94-4e7e-88bb-a730362c7469-openshift-service-ca-bundle\") pod \"sequence-graph-4ff76-59c94c4c8c-4gzqz\" (UID: \"763038c5-1d94-4e7e-88bb-a730362c7469\") " pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz"
May 20 15:55:30.249332 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:30.249297    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/763038c5-1d94-4e7e-88bb-a730362c7469-proxy-tls\") pod \"sequence-graph-4ff76-59c94c4c8c-4gzqz\" (UID: \"763038c5-1d94-4e7e-88bb-a730362c7469\") " pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz"
May 20 15:55:30.251942 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:30.251912    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/763038c5-1d94-4e7e-88bb-a730362c7469-proxy-tls\") pod \"sequence-graph-4ff76-59c94c4c8c-4gzqz\" (UID: \"763038c5-1d94-4e7e-88bb-a730362c7469\") " pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz"
May 20 15:55:30.404228 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:30.404181    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz"
May 20 15:55:30.541496 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:30.539537    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz"]
May 20 15:55:30.715518 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:30.715479    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz" event={"ID":"763038c5-1d94-4e7e-88bb-a730362c7469","Type":"ContainerStarted","Data":"6897e1569eb3ade996c5c65ad0803bdbca3aa92f55ec61c4a4b2e5da7f7d0220"}
May 20 15:55:30.715518 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:30.715516    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz" event={"ID":"763038c5-1d94-4e7e-88bb-a730362c7469","Type":"ContainerStarted","Data":"999946a55cede80bdacd797079e67d19863615889e7c835145dc50c03e413b6e"}
May 20 15:55:30.715743 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:30.715618    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz"
May 20 15:55:30.733368 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:30.733292    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz" podStartSLOduration=1.7332762860000002 podStartE2EDuration="1.733276286s" podCreationTimestamp="2026-05-20 15:55:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:55:30.732479821 +0000 UTC m=+1439.671321377" watchObservedRunningTime="2026-05-20 15:55:30.733276286 +0000 UTC m=+1439.672117854"
May 20 15:55:31.660735 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:31.660697    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c" podUID="13eb4007-0f04-4f0d-8768-7a3de2ab17e4" containerName="ensemble-graph-687f8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:55:31.699629 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:31.699590    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"
May 20 15:55:31.700188 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:31.700158    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c" podUID="022a7db3-b507-4127-9278-3d3308028d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 20 15:55:32.703128 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:32.703099    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"
May 20 15:55:32.703698 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:32.703667    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn" podUID="0b9ffefe-0aad-4ca0-9b52-59b073671f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
May 20 15:55:36.660922 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:36.660882    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c" podUID="13eb4007-0f04-4f0d-8768-7a3de2ab17e4" containerName="ensemble-graph-687f8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:55:36.661330 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:36.660991    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c"
May 20 15:55:36.726082 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:36.726046    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz"
May 20 15:55:39.574086 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:39.574049    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz"]
May 20 15:55:39.574726 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:39.574267    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz" podUID="763038c5-1d94-4e7e-88bb-a730362c7469" containerName="sequence-graph-4ff76" containerID="cri-o://6897e1569eb3ade996c5c65ad0803bdbca3aa92f55ec61c4a4b2e5da7f7d0220" gracePeriod=30
May 20 15:55:39.691449 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:39.691417    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"]
May 20 15:55:39.691715 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:39.691692    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp" podUID="da04dcf0-5363-4b02-b5ba-8e39acd77721" containerName="kserve-container" containerID="cri-o://506926f355d1a0728dc934e0627346ee68a2a0ebb59fb22f3d2e2441dc4b6e62" gracePeriod=30
May 20 15:55:39.691807 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:39.691728    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp" podUID="da04dcf0-5363-4b02-b5ba-8e39acd77721" containerName="kube-rbac-proxy" containerID="cri-o://386540fc48ed1cac7db40815b16a84a49147126586b32bdfa9e3d0077746b9ac" gracePeriod=30
May 20 15:55:39.795466 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:39.795425    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"]
May 20 15:55:39.795768 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:39.795741    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp" podUID="bbffac89-efb6-4741-b082-16c3d625f6b5" containerName="kserve-container" containerID="cri-o://4f9120b1e442a28a473b609246125b45c10b0642249bede65364223e67ee8b5b" gracePeriod=30
May 20 15:55:39.795879 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:39.795778    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp" podUID="bbffac89-efb6-4741-b082-16c3d625f6b5" containerName="kube-rbac-proxy" containerID="cri-o://4b8434e7f16fff2c150f1c9a573152204aa35dffb09cf70c47631c74d644da14" gracePeriod=30
May 20 15:55:39.802793 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:39.802770    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"]
May 20 15:55:39.806805 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:39.806789    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"
May 20 15:55:39.809210 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:39.809180    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-80b6e-predictor-serving-cert\""
May 20 15:55:39.809326 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:39.809265    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-80b6e-kube-rbac-proxy-sar-config\""
May 20 15:55:39.818472 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:39.818450    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"]
May 20 15:55:39.922257 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:39.922226    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"]
May 20 15:55:39.926097 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:39.926073    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"
May 20 15:55:39.928659 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:39.928637    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-80b6e-predictor-serving-cert\""
May 20 15:55:39.928795 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:39.928701    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-80b6e-kube-rbac-proxy-sar-config\""
May 20 15:55:39.935273 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:39.935246    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62f6ac5b-7653-47eb-a82d-c64312fde5ee-proxy-tls\") pod \"success-200-isvc-80b6e-predictor-789b9765b-ldvnz\" (UID: \"62f6ac5b-7653-47eb-a82d-c64312fde5ee\") " pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"
May 20 15:55:39.935377 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:39.935294    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6snf5\" (UniqueName: \"kubernetes.io/projected/62f6ac5b-7653-47eb-a82d-c64312fde5ee-kube-api-access-6snf5\") pod \"success-200-isvc-80b6e-predictor-789b9765b-ldvnz\" (UID: \"62f6ac5b-7653-47eb-a82d-c64312fde5ee\") " pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"
May 20 15:55:39.935420 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:39.935371    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-80b6e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62f6ac5b-7653-47eb-a82d-c64312fde5ee-success-200-isvc-80b6e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-80b6e-predictor-789b9765b-ldvnz\" (UID: \"62f6ac5b-7653-47eb-a82d-c64312fde5ee\") " pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"
May 20 15:55:39.940485 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:39.940463    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"]
May 20 15:55:40.036857 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.036825    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjhnp\" (UniqueName: \"kubernetes.io/projected/0a8d08a2-7d2b-4b3f-9064-8c59b18cab79-kube-api-access-xjhnp\") pod \"error-404-isvc-80b6e-predictor-7f59bbd875-8td9x\" (UID: \"0a8d08a2-7d2b-4b3f-9064-8c59b18cab79\") " pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"
May 20 15:55:40.037048 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.036883    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-80b6e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a8d08a2-7d2b-4b3f-9064-8c59b18cab79-error-404-isvc-80b6e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-80b6e-predictor-7f59bbd875-8td9x\" (UID: \"0a8d08a2-7d2b-4b3f-9064-8c59b18cab79\") " pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"
May 20 15:55:40.037048 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.036946    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62f6ac5b-7653-47eb-a82d-c64312fde5ee-proxy-tls\") pod \"success-200-isvc-80b6e-predictor-789b9765b-ldvnz\" (UID: \"62f6ac5b-7653-47eb-a82d-c64312fde5ee\") " pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"
May 20 15:55:40.037048 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.036978    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6snf5\" (UniqueName: \"kubernetes.io/projected/62f6ac5b-7653-47eb-a82d-c64312fde5ee-kube-api-access-6snf5\") pod \"success-200-isvc-80b6e-predictor-789b9765b-ldvnz\" (UID: \"62f6ac5b-7653-47eb-a82d-c64312fde5ee\") " pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"
May 20 15:55:40.037048 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.037013    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-80b6e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62f6ac5b-7653-47eb-a82d-c64312fde5ee-success-200-isvc-80b6e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-80b6e-predictor-789b9765b-ldvnz\" (UID: \"62f6ac5b-7653-47eb-a82d-c64312fde5ee\") " pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"
May 20 15:55:40.037323 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.037064    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a8d08a2-7d2b-4b3f-9064-8c59b18cab79-proxy-tls\") pod \"error-404-isvc-80b6e-predictor-7f59bbd875-8td9x\" (UID: \"0a8d08a2-7d2b-4b3f-9064-8c59b18cab79\") " pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"
May 20 15:55:40.037774 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.037752    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-80b6e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62f6ac5b-7653-47eb-a82d-c64312fde5ee-success-200-isvc-80b6e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-80b6e-predictor-789b9765b-ldvnz\" (UID: \"62f6ac5b-7653-47eb-a82d-c64312fde5ee\") " pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"
May 20 15:55:40.039633 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.039615    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62f6ac5b-7653-47eb-a82d-c64312fde5ee-proxy-tls\") pod \"success-200-isvc-80b6e-predictor-789b9765b-ldvnz\" (UID: \"62f6ac5b-7653-47eb-a82d-c64312fde5ee\") " pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"
May 20 15:55:40.051525 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.051504    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6snf5\" (UniqueName: \"kubernetes.io/projected/62f6ac5b-7653-47eb-a82d-c64312fde5ee-kube-api-access-6snf5\") pod \"success-200-isvc-80b6e-predictor-789b9765b-ldvnz\" (UID: \"62f6ac5b-7653-47eb-a82d-c64312fde5ee\") " pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"
May 20 15:55:40.120360 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.120339    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"
May 20 15:55:40.138444 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.138421    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-80b6e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a8d08a2-7d2b-4b3f-9064-8c59b18cab79-error-404-isvc-80b6e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-80b6e-predictor-7f59bbd875-8td9x\" (UID: \"0a8d08a2-7d2b-4b3f-9064-8c59b18cab79\") " pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"
May 20 15:55:40.138601 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.138585    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a8d08a2-7d2b-4b3f-9064-8c59b18cab79-proxy-tls\") pod \"error-404-isvc-80b6e-predictor-7f59bbd875-8td9x\" (UID: \"0a8d08a2-7d2b-4b3f-9064-8c59b18cab79\") " pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"
May 20 15:55:40.138694 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.138681    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xjhnp\" (UniqueName: \"kubernetes.io/projected/0a8d08a2-7d2b-4b3f-9064-8c59b18cab79-kube-api-access-xjhnp\") pod \"error-404-isvc-80b6e-predictor-7f59bbd875-8td9x\" (UID: \"0a8d08a2-7d2b-4b3f-9064-8c59b18cab79\") " pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"
May 20 15:55:40.139114 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.139096    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-80b6e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a8d08a2-7d2b-4b3f-9064-8c59b18cab79-error-404-isvc-80b6e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-80b6e-predictor-7f59bbd875-8td9x\" (UID: \"0a8d08a2-7d2b-4b3f-9064-8c59b18cab79\") " pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"
May 20 15:55:40.141181 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.141162    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a8d08a2-7d2b-4b3f-9064-8c59b18cab79-proxy-tls\") pod \"error-404-isvc-80b6e-predictor-7f59bbd875-8td9x\" (UID: \"0a8d08a2-7d2b-4b3f-9064-8c59b18cab79\") " pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"
May 20 15:55:40.150134 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.150116    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjhnp\" (UniqueName: \"kubernetes.io/projected/0a8d08a2-7d2b-4b3f-9064-8c59b18cab79-kube-api-access-xjhnp\") pod \"error-404-isvc-80b6e-predictor-7f59bbd875-8td9x\" (UID: \"0a8d08a2-7d2b-4b3f-9064-8c59b18cab79\") " pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"
May 20 15:55:40.237083 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.237053    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"
May 20 15:55:40.257768 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.257744    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"]
May 20 15:55:40.382872 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.382847    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"]
May 20 15:55:40.753001 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.752923    2568 generic.go:358] "Generic (PLEG): container finished" podID="da04dcf0-5363-4b02-b5ba-8e39acd77721" containerID="386540fc48ed1cac7db40815b16a84a49147126586b32bdfa9e3d0077746b9ac" exitCode=2
May 20 15:55:40.753411 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.752999    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp" event={"ID":"da04dcf0-5363-4b02-b5ba-8e39acd77721","Type":"ContainerDied","Data":"386540fc48ed1cac7db40815b16a84a49147126586b32bdfa9e3d0077746b9ac"}
May 20 15:55:40.754564 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.754536    2568 generic.go:358] "Generic (PLEG): container finished" podID="bbffac89-efb6-4741-b082-16c3d625f6b5" containerID="4b8434e7f16fff2c150f1c9a573152204aa35dffb09cf70c47631c74d644da14" exitCode=2
May 20 15:55:40.754702 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.754614    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp" event={"ID":"bbffac89-efb6-4741-b082-16c3d625f6b5","Type":"ContainerDied","Data":"4b8434e7f16fff2c150f1c9a573152204aa35dffb09cf70c47631c74d644da14"}
May 20 15:55:40.756048 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.756007    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x" event={"ID":"0a8d08a2-7d2b-4b3f-9064-8c59b18cab79","Type":"ContainerStarted","Data":"33b517e82dd506f007822f624f99412b7966b262b738c4ffb7b1a19b03253a6f"}
May 20 15:55:40.756143 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.756048    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x" event={"ID":"0a8d08a2-7d2b-4b3f-9064-8c59b18cab79","Type":"ContainerStarted","Data":"fea7ee8e11233e23eac51f8faea7398b1fd4e87a64c2a8ce0eb2073bbfbd117f"}
May 20 15:55:40.756143 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.756063    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x" event={"ID":"0a8d08a2-7d2b-4b3f-9064-8c59b18cab79","Type":"ContainerStarted","Data":"f35da4033b944b1952c5f1369123a74559787a0a535d0fe9d3aab785da440fea"}
May 20 15:55:40.756246 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.756151    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"
May 20 15:55:40.757461 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.757438    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz" event={"ID":"62f6ac5b-7653-47eb-a82d-c64312fde5ee","Type":"ContainerStarted","Data":"e0b74451a2f48bc7924afbda411451f4fbeee866e29c91f8168e14a5a7e40f26"}
May 20 15:55:40.757547 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.757468    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz" event={"ID":"62f6ac5b-7653-47eb-a82d-c64312fde5ee","Type":"ContainerStarted","Data":"dd44e1460b642e45f1298f5b5dbe27bbee80c484810a0fec246cf50573dee7d8"}
May 20 15:55:40.757547 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.757478    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz" event={"ID":"62f6ac5b-7653-47eb-a82d-c64312fde5ee","Type":"ContainerStarted","Data":"fd384be36fd99b519b62e21f8fa7155f49e86a385b1e745229e961cdd9814edc"}
May 20 15:55:40.757640 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.757582    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"
May 20 15:55:40.776627 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.776572    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x" podStartSLOduration=1.7765550669999999 podStartE2EDuration="1.776555067s" podCreationTimestamp="2026-05-20 15:55:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:55:40.775337005 +0000 UTC m=+1449.714178561" watchObservedRunningTime="2026-05-20 15:55:40.776555067 +0000 UTC m=+1449.715396628"
May 20 15:55:40.796590 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:40.796492    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz" podStartSLOduration=1.796479749 podStartE2EDuration="1.796479749s" podCreationTimestamp="2026-05-20 15:55:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:55:40.794996554 +0000 UTC m=+1449.733838113" watchObservedRunningTime="2026-05-20 15:55:40.796479749 +0000 UTC m=+1449.735321305"
May 20 15:55:41.660782 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:41.660745    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c" podUID="13eb4007-0f04-4f0d-8768-7a3de2ab17e4" containerName="ensemble-graph-687f8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:55:41.700984 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:41.700950    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c" podUID="022a7db3-b507-4127-9278-3d3308028d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 20 15:55:41.723495 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:41.723464    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz" podUID="763038c5-1d94-4e7e-88bb-a730362c7469" containerName="sequence-graph-4ff76" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:55:41.760590 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:41.760561    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"
May 20 15:55:41.760590 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:41.760598    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"
May 20 15:55:41.761542 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:41.761517    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x" podUID="0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 20 15:55:41.761608 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:41.761534    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz" podUID="62f6ac5b-7653-47eb-a82d-c64312fde5ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
May 20 15:55:42.485066 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:42.484996    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp" podUID="da04dcf0-5363-4b02-b5ba-8e39acd77721" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
May 20 15:55:42.703816 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:42.703775    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn" podUID="0b9ffefe-0aad-4ca0-9b52-59b073671f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
May 20 15:55:42.766866 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:42.766770    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x" podUID="0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 20 15:55:42.767240 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:42.766889    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz" podUID="62f6ac5b-7653-47eb-a82d-c64312fde5ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
May 20 15:55:43.055111 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.055082    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"
May 20 15:55:43.065996 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.065976    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz7rl\" (UniqueName: \"kubernetes.io/projected/da04dcf0-5363-4b02-b5ba-8e39acd77721-kube-api-access-sz7rl\") pod \"da04dcf0-5363-4b02-b5ba-8e39acd77721\" (UID: \"da04dcf0-5363-4b02-b5ba-8e39acd77721\") "
May 20 15:55:43.066117 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.066044    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da04dcf0-5363-4b02-b5ba-8e39acd77721-proxy-tls\") pod \"da04dcf0-5363-4b02-b5ba-8e39acd77721\" (UID: \"da04dcf0-5363-4b02-b5ba-8e39acd77721\") "
May 20 15:55:43.068042 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.067989    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da04dcf0-5363-4b02-b5ba-8e39acd77721-kube-api-access-sz7rl" (OuterVolumeSpecName: "kube-api-access-sz7rl") pod "da04dcf0-5363-4b02-b5ba-8e39acd77721" (UID: "da04dcf0-5363-4b02-b5ba-8e39acd77721"). InnerVolumeSpecName "kube-api-access-sz7rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:55:43.068185 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.068164    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da04dcf0-5363-4b02-b5ba-8e39acd77721-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "da04dcf0-5363-4b02-b5ba-8e39acd77721" (UID: "da04dcf0-5363-4b02-b5ba-8e39acd77721"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:55:43.166780 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.166742    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-4ff76-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da04dcf0-5363-4b02-b5ba-8e39acd77721-error-404-isvc-4ff76-kube-rbac-proxy-sar-config\") pod \"da04dcf0-5363-4b02-b5ba-8e39acd77721\" (UID: \"da04dcf0-5363-4b02-b5ba-8e39acd77721\") "
May 20 15:55:43.166986 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.166890    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da04dcf0-5363-4b02-b5ba-8e39acd77721-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:55:43.166986 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.166902    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz7rl\" (UniqueName: \"kubernetes.io/projected/da04dcf0-5363-4b02-b5ba-8e39acd77721-kube-api-access-sz7rl\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:55:43.167227 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.167201    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da04dcf0-5363-4b02-b5ba-8e39acd77721-error-404-isvc-4ff76-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-4ff76-kube-rbac-proxy-sar-config") pod "da04dcf0-5363-4b02-b5ba-8e39acd77721" (UID: "da04dcf0-5363-4b02-b5ba-8e39acd77721"). InnerVolumeSpecName "error-404-isvc-4ff76-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:55:43.267371 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.267330    2568 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-4ff76-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da04dcf0-5363-4b02-b5ba-8e39acd77721-error-404-isvc-4ff76-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:55:43.489598 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.489563    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp" podUID="bbffac89-efb6-4741-b082-16c3d625f6b5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
May 20 15:55:43.649181 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.649157    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"
May 20 15:55:43.672056 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.670481    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-4ff76-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbffac89-efb6-4741-b082-16c3d625f6b5-success-200-isvc-4ff76-kube-rbac-proxy-sar-config\") pod \"bbffac89-efb6-4741-b082-16c3d625f6b5\" (UID: \"bbffac89-efb6-4741-b082-16c3d625f6b5\") "
May 20 15:55:43.672056 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.670534    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxzt6\" (UniqueName: \"kubernetes.io/projected/bbffac89-efb6-4741-b082-16c3d625f6b5-kube-api-access-dxzt6\") pod \"bbffac89-efb6-4741-b082-16c3d625f6b5\" (UID: \"bbffac89-efb6-4741-b082-16c3d625f6b5\") "
May 20 15:55:43.672056 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.670638    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbffac89-efb6-4741-b082-16c3d625f6b5-proxy-tls\") pod \"bbffac89-efb6-4741-b082-16c3d625f6b5\" (UID: \"bbffac89-efb6-4741-b082-16c3d625f6b5\") "
May 20 15:55:43.672056 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.671155    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbffac89-efb6-4741-b082-16c3d625f6b5-success-200-isvc-4ff76-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-4ff76-kube-rbac-proxy-sar-config") pod "bbffac89-efb6-4741-b082-16c3d625f6b5" (UID: "bbffac89-efb6-4741-b082-16c3d625f6b5"). InnerVolumeSpecName "success-200-isvc-4ff76-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:55:43.673837 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.673461    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbffac89-efb6-4741-b082-16c3d625f6b5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bbffac89-efb6-4741-b082-16c3d625f6b5" (UID: "bbffac89-efb6-4741-b082-16c3d625f6b5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:55:43.673837 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.673782    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbffac89-efb6-4741-b082-16c3d625f6b5-kube-api-access-dxzt6" (OuterVolumeSpecName: "kube-api-access-dxzt6") pod "bbffac89-efb6-4741-b082-16c3d625f6b5" (UID: "bbffac89-efb6-4741-b082-16c3d625f6b5"). InnerVolumeSpecName "kube-api-access-dxzt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:55:43.771531 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.771421    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbffac89-efb6-4741-b082-16c3d625f6b5-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:55:43.771531 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.771457    2568 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-4ff76-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbffac89-efb6-4741-b082-16c3d625f6b5-success-200-isvc-4ff76-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:55:43.771531 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.771474    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxzt6\" (UniqueName: \"kubernetes.io/projected/bbffac89-efb6-4741-b082-16c3d625f6b5-kube-api-access-dxzt6\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:55:43.772923 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.772893    2568 generic.go:358] "Generic (PLEG): container finished" podID="da04dcf0-5363-4b02-b5ba-8e39acd77721" containerID="506926f355d1a0728dc934e0627346ee68a2a0ebb59fb22f3d2e2441dc4b6e62" exitCode=0
May 20 15:55:43.773064 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.773006    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp" event={"ID":"da04dcf0-5363-4b02-b5ba-8e39acd77721","Type":"ContainerDied","Data":"506926f355d1a0728dc934e0627346ee68a2a0ebb59fb22f3d2e2441dc4b6e62"}
May 20 15:55:43.773152 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.773075    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp" event={"ID":"da04dcf0-5363-4b02-b5ba-8e39acd77721","Type":"ContainerDied","Data":"1079a8636ab0dd4e7f908c93c11547e5674c8c1859385170c3283cd6765f2f99"}
May 20 15:55:43.773152 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.773080    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"
May 20 15:55:43.773152 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.773107    2568 scope.go:117] "RemoveContainer" containerID="386540fc48ed1cac7db40815b16a84a49147126586b32bdfa9e3d0077746b9ac"
May 20 15:55:43.774683 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.774665    2568 generic.go:358] "Generic (PLEG): container finished" podID="bbffac89-efb6-4741-b082-16c3d625f6b5" containerID="4f9120b1e442a28a473b609246125b45c10b0642249bede65364223e67ee8b5b" exitCode=0
May 20 15:55:43.774753 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.774710    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp" event={"ID":"bbffac89-efb6-4741-b082-16c3d625f6b5","Type":"ContainerDied","Data":"4f9120b1e442a28a473b609246125b45c10b0642249bede65364223e67ee8b5b"}
May 20 15:55:43.774753 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.774738    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp" event={"ID":"bbffac89-efb6-4741-b082-16c3d625f6b5","Type":"ContainerDied","Data":"dbfde16948983f994b00708539d4e51eb60d5b651c1c460fb26311920148aac0"}
May 20 15:55:43.774867 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.774824    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"
May 20 15:55:43.783089 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.783070    2568 scope.go:117] "RemoveContainer" containerID="506926f355d1a0728dc934e0627346ee68a2a0ebb59fb22f3d2e2441dc4b6e62"
May 20 15:55:43.791479 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.791464    2568 scope.go:117] "RemoveContainer" containerID="386540fc48ed1cac7db40815b16a84a49147126586b32bdfa9e3d0077746b9ac"
May 20 15:55:43.791721 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:55:43.791703    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386540fc48ed1cac7db40815b16a84a49147126586b32bdfa9e3d0077746b9ac\": container with ID starting with 386540fc48ed1cac7db40815b16a84a49147126586b32bdfa9e3d0077746b9ac not found: ID does not exist" containerID="386540fc48ed1cac7db40815b16a84a49147126586b32bdfa9e3d0077746b9ac"
May 20 15:55:43.791800 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.791737    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386540fc48ed1cac7db40815b16a84a49147126586b32bdfa9e3d0077746b9ac"} err="failed to get container status \"386540fc48ed1cac7db40815b16a84a49147126586b32bdfa9e3d0077746b9ac\": rpc error: code = NotFound desc = could not find container \"386540fc48ed1cac7db40815b16a84a49147126586b32bdfa9e3d0077746b9ac\": container with ID starting with 386540fc48ed1cac7db40815b16a84a49147126586b32bdfa9e3d0077746b9ac not found: ID does not exist"
May 20 15:55:43.791800 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.791763    2568 scope.go:117] "RemoveContainer" containerID="506926f355d1a0728dc934e0627346ee68a2a0ebb59fb22f3d2e2441dc4b6e62"
May 20 15:55:43.792082 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:55:43.792009    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"506926f355d1a0728dc934e0627346ee68a2a0ebb59fb22f3d2e2441dc4b6e62\": container with ID starting with 506926f355d1a0728dc934e0627346ee68a2a0ebb59fb22f3d2e2441dc4b6e62 not found: ID does not exist" containerID="506926f355d1a0728dc934e0627346ee68a2a0ebb59fb22f3d2e2441dc4b6e62"
May 20 15:55:43.792130 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.792108    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506926f355d1a0728dc934e0627346ee68a2a0ebb59fb22f3d2e2441dc4b6e62"} err="failed to get container status \"506926f355d1a0728dc934e0627346ee68a2a0ebb59fb22f3d2e2441dc4b6e62\": rpc error: code = NotFound desc = could not find container \"506926f355d1a0728dc934e0627346ee68a2a0ebb59fb22f3d2e2441dc4b6e62\": container with ID starting with 506926f355d1a0728dc934e0627346ee68a2a0ebb59fb22f3d2e2441dc4b6e62 not found: ID does not exist"
May 20 15:55:43.792197 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.792133    2568 scope.go:117] "RemoveContainer" containerID="4b8434e7f16fff2c150f1c9a573152204aa35dffb09cf70c47631c74d644da14"
May 20 15:55:43.795827 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.795802    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"]
May 20 15:55:43.797741 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.797722    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4ff76-predictor-658c9d7c5-bnlbp"]
May 20 15:55:43.801124 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.801111    2568 scope.go:117] "RemoveContainer" containerID="4f9120b1e442a28a473b609246125b45c10b0642249bede65364223e67ee8b5b"
May 20 15:55:43.805503 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.805481    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"]
May 20 15:55:43.808278 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.808257    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4ff76-predictor-5878d445d4-msxvp"]
May 20 15:55:43.809905 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.809877    2568 scope.go:117] "RemoveContainer" containerID="4b8434e7f16fff2c150f1c9a573152204aa35dffb09cf70c47631c74d644da14"
May 20 15:55:43.810161 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:55:43.810144    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b8434e7f16fff2c150f1c9a573152204aa35dffb09cf70c47631c74d644da14\": container with ID starting with 4b8434e7f16fff2c150f1c9a573152204aa35dffb09cf70c47631c74d644da14 not found: ID does not exist" containerID="4b8434e7f16fff2c150f1c9a573152204aa35dffb09cf70c47631c74d644da14"
May 20 15:55:43.810210 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.810168    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b8434e7f16fff2c150f1c9a573152204aa35dffb09cf70c47631c74d644da14"} err="failed to get container status \"4b8434e7f16fff2c150f1c9a573152204aa35dffb09cf70c47631c74d644da14\": rpc error: code = NotFound desc = could not find container \"4b8434e7f16fff2c150f1c9a573152204aa35dffb09cf70c47631c74d644da14\": container with ID starting with 4b8434e7f16fff2c150f1c9a573152204aa35dffb09cf70c47631c74d644da14 not found: ID does not exist"
May 20 15:55:43.810210 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.810189    2568 scope.go:117] "RemoveContainer" containerID="4f9120b1e442a28a473b609246125b45c10b0642249bede65364223e67ee8b5b"
May 20 15:55:43.810387 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:55:43.810369    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f9120b1e442a28a473b609246125b45c10b0642249bede65364223e67ee8b5b\": container with ID starting with 4f9120b1e442a28a473b609246125b45c10b0642249bede65364223e67ee8b5b not found: ID does not exist" containerID="4f9120b1e442a28a473b609246125b45c10b0642249bede65364223e67ee8b5b"
May 20 15:55:43.810433 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:43.810391    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f9120b1e442a28a473b609246125b45c10b0642249bede65364223e67ee8b5b"} err="failed to get container status \"4f9120b1e442a28a473b609246125b45c10b0642249bede65364223e67ee8b5b\": rpc error: code = NotFound desc = could not find container \"4f9120b1e442a28a473b609246125b45c10b0642249bede65364223e67ee8b5b\": container with ID starting with 4f9120b1e442a28a473b609246125b45c10b0642249bede65364223e67ee8b5b not found: ID does not exist"
May 20 15:55:45.643661 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:45.643629    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbffac89-efb6-4741-b082-16c3d625f6b5" path="/var/lib/kubelet/pods/bbffac89-efb6-4741-b082-16c3d625f6b5/volumes"
May 20 15:55:45.644087 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:45.644071    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da04dcf0-5363-4b02-b5ba-8e39acd77721" path="/var/lib/kubelet/pods/da04dcf0-5363-4b02-b5ba-8e39acd77721/volumes"
May 20 15:55:46.660931 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:46.660898    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c" podUID="13eb4007-0f04-4f0d-8768-7a3de2ab17e4" containerName="ensemble-graph-687f8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:55:46.723939 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:46.723889    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz" podUID="763038c5-1d94-4e7e-88bb-a730362c7469" containerName="sequence-graph-4ff76" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:55:47.771966 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:47.771923    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"
May 20 15:55:47.772820 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:47.772790    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz" podUID="62f6ac5b-7653-47eb-a82d-c64312fde5ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
May 20 15:55:47.773154 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:47.773130    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"
May 20 15:55:47.773641 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:47.773618    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x" podUID="0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 20 15:55:51.660732 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:51.660691    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c" podUID="13eb4007-0f04-4f0d-8768-7a3de2ab17e4" containerName="ensemble-graph-687f8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:55:51.700403 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:51.700370    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c" podUID="022a7db3-b507-4127-9278-3d3308028d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 20 15:55:51.724088 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:51.724053    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz" podUID="763038c5-1d94-4e7e-88bb-a730362c7469" containerName="sequence-graph-4ff76" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:55:51.724242 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:51.724164    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz"
May 20 15:55:52.703609 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:52.703574    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn" podUID="0b9ffefe-0aad-4ca0-9b52-59b073671f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
May 20 15:55:53.849617 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:55:53.849566    2568 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice/crio-386540fc48ed1cac7db40815b16a84a49147126586b32bdfa9e3d0077746b9ac.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbffac89_efb6_4741_b082_16c3d625f6b5.slice/crio-4b8434e7f16fff2c150f1c9a573152204aa35dffb09cf70c47631c74d644da14.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbffac89_efb6_4741_b082_16c3d625f6b5.slice/crio-4f9120b1e442a28a473b609246125b45c10b0642249bede65364223e67ee8b5b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice/crio-conmon-386540fc48ed1cac7db40815b16a84a49147126586b32bdfa9e3d0077746b9ac.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice/crio-1079a8636ab0dd4e7f908c93c11547e5674c8c1859385170c3283cd6765f2f99\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice/crio-506926f355d1a0728dc934e0627346ee68a2a0ebb59fb22f3d2e2441dc4b6e62.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbffac89_efb6_4741_b082_16c3d625f6b5.slice/crio-conmon-4b8434e7f16fff2c150f1c9a573152204aa35dffb09cf70c47631c74d644da14.scope\": RecentStats: unable to find data in memory cache]"
May 20 15:55:53.849989 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:55:53.849596    2568 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice/crio-506926f355d1a0728dc934e0627346ee68a2a0ebb59fb22f3d2e2441dc4b6e62.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbffac89_efb6_4741_b082_16c3d625f6b5.slice/crio-conmon-4f9120b1e442a28a473b609246125b45c10b0642249bede65364223e67ee8b5b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbffac89_efb6_4741_b082_16c3d625f6b5.slice/crio-dbfde16948983f994b00708539d4e51eb60d5b651c1c460fb26311920148aac0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbffac89_efb6_4741_b082_16c3d625f6b5.slice/crio-4b8434e7f16fff2c150f1c9a573152204aa35dffb09cf70c47631c74d644da14.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbffac89_efb6_4741_b082_16c3d625f6b5.slice/crio-conmon-4b8434e7f16fff2c150f1c9a573152204aa35dffb09cf70c47631c74d644da14.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13eb4007_0f04_4f0d_8768_7a3de2ab17e4.slice/crio-3fb38dec2151ae7ac659bf30c318c6f42d8832883b192681f5ec19a128a4da75.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice/crio-conmon-386540fc48ed1cac7db40815b16a84a49147126586b32bdfa9e3d0077746b9ac.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice/crio-386540fc48ed1cac7db40815b16a84a49147126586b32bdfa9e3d0077746b9ac.scope\": RecentStats: unable to find data in memory cache]"
May 20 15:55:53.849989 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:55:53.849787    2568 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice/crio-386540fc48ed1cac7db40815b16a84a49147126586b32bdfa9e3d0077746b9ac.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice/crio-1079a8636ab0dd4e7f908c93c11547e5674c8c1859385170c3283cd6765f2f99\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice/crio-506926f355d1a0728dc934e0627346ee68a2a0ebb59fb22f3d2e2441dc4b6e62.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbffac89_efb6_4741_b082_16c3d625f6b5.slice/crio-4b8434e7f16fff2c150f1c9a573152204aa35dffb09cf70c47631c74d644da14.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbffac89_efb6_4741_b082_16c3d625f6b5.slice/crio-conmon-4b8434e7f16fff2c150f1c9a573152204aa35dffb09cf70c47631c74d644da14.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13eb4007_0f04_4f0d_8768_7a3de2ab17e4.slice/crio-conmon-3fb38dec2151ae7ac659bf30c318c6f42d8832883b192681f5ec19a128a4da75.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice/crio-conmon-386540fc48ed1cac7db40815b16a84a49147126586b32bdfa9e3d0077746b9ac.scope\": RecentStats: unable to find data in memory cache]"
May 20 15:55:53.850205 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:55:53.849883    2568 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbffac89_efb6_4741_b082_16c3d625f6b5.slice/crio-4b8434e7f16fff2c150f1c9a573152204aa35dffb09cf70c47631c74d644da14.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbffac89_efb6_4741_b082_16c3d625f6b5.slice/crio-conmon-4b8434e7f16fff2c150f1c9a573152204aa35dffb09cf70c47631c74d644da14.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice/crio-conmon-386540fc48ed1cac7db40815b16a84a49147126586b32bdfa9e3d0077746b9ac.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice/crio-506926f355d1a0728dc934e0627346ee68a2a0ebb59fb22f3d2e2441dc4b6e62.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice/crio-386540fc48ed1cac7db40815b16a84a49147126586b32bdfa9e3d0077746b9ac.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13eb4007_0f04_4f0d_8768_7a3de2ab17e4.slice/crio-3fb38dec2151ae7ac659bf30c318c6f42d8832883b192681f5ec19a128a4da75.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13eb4007_0f04_4f0d_8768_7a3de2ab17e4.slice/crio-conmon-3fb38dec2151ae7ac659bf30c318c6f42d8832883b192681f5ec19a128a4da75.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbffac89_efb6_4741_b082_16c3d625f6b5.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice/crio-1079a8636ab0dd4e7f908c93c11547e5674c8c1859385170c3283cd6765f2f99\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbffac89_efb6_4741_b082_16c3d625f6b5.slice/crio-4f9120b1e442a28a473b609246125b45c10b0642249bede65364223e67ee8b5b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice/crio-conmon-506926f355d1a0728dc934e0627346ee68a2a0ebb59fb22f3d2e2441dc4b6e62.scope\": RecentStats: unable to find data in memory cache]"
May 20 15:55:53.850205 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:55:53.850087    2568 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbffac89_efb6_4741_b082_16c3d625f6b5.slice/crio-conmon-4f9120b1e442a28a473b609246125b45c10b0642249bede65364223e67ee8b5b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice/crio-conmon-506926f355d1a0728dc934e0627346ee68a2a0ebb59fb22f3d2e2441dc4b6e62.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbffac89_efb6_4741_b082_16c3d625f6b5.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice/crio-1079a8636ab0dd4e7f908c93c11547e5674c8c1859385170c3283cd6765f2f99\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbffac89_efb6_4741_b082_16c3d625f6b5.slice/crio-4f9120b1e442a28a473b609246125b45c10b0642249bede65364223e67ee8b5b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13eb4007_0f04_4f0d_8768_7a3de2ab17e4.slice/crio-3fb38dec2151ae7ac659bf30c318c6f42d8832883b192681f5ec19a128a4da75.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbffac89_efb6_4741_b082_16c3d625f6b5.slice/crio-dbfde16948983f994b00708539d4e51eb60d5b651c1c460fb26311920148aac0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dcf0_5363_4b02_b5ba_8e39acd77721.slice/crio-506926f355d1a0728dc934e0627346ee68a2a0ebb59fb22f3d2e2441dc4b6e62.scope\": RecentStats: unable to find data in memory cache]"
May 20 15:55:54.485973 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:54.485949    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c"
May 20 15:55:54.565426 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:54.565393    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13eb4007-0f04-4f0d-8768-7a3de2ab17e4-openshift-service-ca-bundle\") pod \"13eb4007-0f04-4f0d-8768-7a3de2ab17e4\" (UID: \"13eb4007-0f04-4f0d-8768-7a3de2ab17e4\") "
May 20 15:55:54.565426 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:54.565427    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13eb4007-0f04-4f0d-8768-7a3de2ab17e4-proxy-tls\") pod \"13eb4007-0f04-4f0d-8768-7a3de2ab17e4\" (UID: \"13eb4007-0f04-4f0d-8768-7a3de2ab17e4\") "
May 20 15:55:54.565730 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:54.565707    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13eb4007-0f04-4f0d-8768-7a3de2ab17e4-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "13eb4007-0f04-4f0d-8768-7a3de2ab17e4" (UID: "13eb4007-0f04-4f0d-8768-7a3de2ab17e4"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:55:54.567594 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:54.567569    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13eb4007-0f04-4f0d-8768-7a3de2ab17e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "13eb4007-0f04-4f0d-8768-7a3de2ab17e4" (UID: "13eb4007-0f04-4f0d-8768-7a3de2ab17e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:55:54.666967 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:54.666894    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13eb4007-0f04-4f0d-8768-7a3de2ab17e4-openshift-service-ca-bundle\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:55:54.666967 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:54.666921    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13eb4007-0f04-4f0d-8768-7a3de2ab17e4-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:55:54.814859 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:54.814825    2568 generic.go:358] "Generic (PLEG): container finished" podID="13eb4007-0f04-4f0d-8768-7a3de2ab17e4" containerID="3fb38dec2151ae7ac659bf30c318c6f42d8832883b192681f5ec19a128a4da75" exitCode=0
May 20 15:55:54.815046 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:54.814882    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c" event={"ID":"13eb4007-0f04-4f0d-8768-7a3de2ab17e4","Type":"ContainerDied","Data":"3fb38dec2151ae7ac659bf30c318c6f42d8832883b192681f5ec19a128a4da75"}
May 20 15:55:54.815046 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:54.814884    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c"
May 20 15:55:54.815046 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:54.814920    2568 scope.go:117] "RemoveContainer" containerID="3fb38dec2151ae7ac659bf30c318c6f42d8832883b192681f5ec19a128a4da75"
May 20 15:55:54.815046 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:54.814910    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c" event={"ID":"13eb4007-0f04-4f0d-8768-7a3de2ab17e4","Type":"ContainerDied","Data":"8882501e059416abaefda6a3080f2aa9a5a52e153e7a38b26115eba5aa0b1a1e"}
May 20 15:55:54.825764 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:54.825738    2568 scope.go:117] "RemoveContainer" containerID="3fb38dec2151ae7ac659bf30c318c6f42d8832883b192681f5ec19a128a4da75"
May 20 15:55:54.826242 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:55:54.826219    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb38dec2151ae7ac659bf30c318c6f42d8832883b192681f5ec19a128a4da75\": container with ID starting with 3fb38dec2151ae7ac659bf30c318c6f42d8832883b192681f5ec19a128a4da75 not found: ID does not exist" containerID="3fb38dec2151ae7ac659bf30c318c6f42d8832883b192681f5ec19a128a4da75"
May 20 15:55:54.826336 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:54.826249    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb38dec2151ae7ac659bf30c318c6f42d8832883b192681f5ec19a128a4da75"} err="failed to get container status \"3fb38dec2151ae7ac659bf30c318c6f42d8832883b192681f5ec19a128a4da75\": rpc error: code = NotFound desc = could not find container \"3fb38dec2151ae7ac659bf30c318c6f42d8832883b192681f5ec19a128a4da75\": container with ID starting with 3fb38dec2151ae7ac659bf30c318c6f42d8832883b192681f5ec19a128a4da75 not found: ID does not exist"
May 20 15:55:54.836014 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:54.835989    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c"]
May 20 15:55:54.840483 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:54.840463    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-687f8-5b9ff78587-xmm2c"]
May 20 15:55:55.644233 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:55.644204    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13eb4007-0f04-4f0d-8768-7a3de2ab17e4" path="/var/lib/kubelet/pods/13eb4007-0f04-4f0d-8768-7a3de2ab17e4/volumes"
May 20 15:55:56.723642 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:56.723594    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz" podUID="763038c5-1d94-4e7e-88bb-a730362c7469" containerName="sequence-graph-4ff76" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:55:57.773500 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:57.773454    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz" podUID="62f6ac5b-7653-47eb-a82d-c64312fde5ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
May 20 15:55:57.773999 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:55:57.773666    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x" podUID="0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 20 15:56:01.700206 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:01.700166    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c" podUID="022a7db3-b507-4127-9278-3d3308028d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 20 15:56:01.723504 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:01.723474    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz" podUID="763038c5-1d94-4e7e-88bb-a730362c7469" containerName="sequence-graph-4ff76" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:56:02.704453 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:02.704414    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn" podUID="0b9ffefe-0aad-4ca0-9b52-59b073671f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
May 20 15:56:06.723960 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:06.723907    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz" podUID="763038c5-1d94-4e7e-88bb-a730362c7469" containerName="sequence-graph-4ff76" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:56:07.773266 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:07.773215    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz" podUID="62f6ac5b-7653-47eb-a82d-c64312fde5ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
May 20 15:56:07.773738 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:07.773633    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x" podUID="0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 20 15:56:09.728203 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:09.728176    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz"
May 20 15:56:09.804337 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:09.804298    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/763038c5-1d94-4e7e-88bb-a730362c7469-proxy-tls\") pod \"763038c5-1d94-4e7e-88bb-a730362c7469\" (UID: \"763038c5-1d94-4e7e-88bb-a730362c7469\") "
May 20 15:56:09.804489 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:09.804359    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/763038c5-1d94-4e7e-88bb-a730362c7469-openshift-service-ca-bundle\") pod \"763038c5-1d94-4e7e-88bb-a730362c7469\" (UID: \"763038c5-1d94-4e7e-88bb-a730362c7469\") "
May 20 15:56:09.804711 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:09.804691    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/763038c5-1d94-4e7e-88bb-a730362c7469-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "763038c5-1d94-4e7e-88bb-a730362c7469" (UID: "763038c5-1d94-4e7e-88bb-a730362c7469"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:56:09.806609 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:09.806571    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/763038c5-1d94-4e7e-88bb-a730362c7469-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "763038c5-1d94-4e7e-88bb-a730362c7469" (UID: "763038c5-1d94-4e7e-88bb-a730362c7469"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:56:09.869266 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:09.869056    2568 generic.go:358] "Generic (PLEG): container finished" podID="763038c5-1d94-4e7e-88bb-a730362c7469" containerID="6897e1569eb3ade996c5c65ad0803bdbca3aa92f55ec61c4a4b2e5da7f7d0220" exitCode=0
May 20 15:56:09.869266 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:09.869158    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz" event={"ID":"763038c5-1d94-4e7e-88bb-a730362c7469","Type":"ContainerDied","Data":"6897e1569eb3ade996c5c65ad0803bdbca3aa92f55ec61c4a4b2e5da7f7d0220"}
May 20 15:56:09.869266 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:09.869175    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz"
May 20 15:56:09.869266 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:09.869189    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz" event={"ID":"763038c5-1d94-4e7e-88bb-a730362c7469","Type":"ContainerDied","Data":"999946a55cede80bdacd797079e67d19863615889e7c835145dc50c03e413b6e"}
May 20 15:56:09.869266 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:09.869211    2568 scope.go:117] "RemoveContainer" containerID="6897e1569eb3ade996c5c65ad0803bdbca3aa92f55ec61c4a4b2e5da7f7d0220"
May 20 15:56:09.881161 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:09.881007    2568 scope.go:117] "RemoveContainer" containerID="6897e1569eb3ade996c5c65ad0803bdbca3aa92f55ec61c4a4b2e5da7f7d0220"
May 20 15:56:09.881427 ip-10-0-129-117 kubenswrapper[2568]: E0520 15:56:09.881402    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6897e1569eb3ade996c5c65ad0803bdbca3aa92f55ec61c4a4b2e5da7f7d0220\": container with ID starting with 6897e1569eb3ade996c5c65ad0803bdbca3aa92f55ec61c4a4b2e5da7f7d0220 not found: ID does not exist" containerID="6897e1569eb3ade996c5c65ad0803bdbca3aa92f55ec61c4a4b2e5da7f7d0220"
May 20 15:56:09.881518 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:09.881440    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6897e1569eb3ade996c5c65ad0803bdbca3aa92f55ec61c4a4b2e5da7f7d0220"} err="failed to get container status \"6897e1569eb3ade996c5c65ad0803bdbca3aa92f55ec61c4a4b2e5da7f7d0220\": rpc error: code = NotFound desc = could not find container \"6897e1569eb3ade996c5c65ad0803bdbca3aa92f55ec61c4a4b2e5da7f7d0220\": container with ID starting with 6897e1569eb3ade996c5c65ad0803bdbca3aa92f55ec61c4a4b2e5da7f7d0220 not found: ID does not exist"
May 20 15:56:09.901331 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:09.901299    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz"]
May 20 15:56:09.902932 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:09.902910    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4ff76-59c94c4c8c-4gzqz"]
May 20 15:56:09.905749 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:09.905726    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/763038c5-1d94-4e7e-88bb-a730362c7469-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:56:09.905749 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:09.905750    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/763038c5-1d94-4e7e-88bb-a730362c7469-openshift-service-ca-bundle\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 15:56:11.651147 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:11.651111    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="763038c5-1d94-4e7e-88bb-a730362c7469" path="/var/lib/kubelet/pods/763038c5-1d94-4e7e-88bb-a730362c7469/volumes"
May 20 15:56:11.700426 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:11.700380    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c" podUID="022a7db3-b507-4127-9278-3d3308028d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 20 15:56:12.704653 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:12.704622    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"
May 20 15:56:17.773107 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:17.773055    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz" podUID="62f6ac5b-7653-47eb-a82d-c64312fde5ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
May 20 15:56:17.773554 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:17.773531    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x" podUID="0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 20 15:56:21.700731 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:21.700699    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"
May 20 15:56:27.773663 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:27.773608    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz" podUID="62f6ac5b-7653-47eb-a82d-c64312fde5ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
May 20 15:56:27.774180 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:27.773608    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x" podUID="0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 20 15:56:31.672634 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:31.672595    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 15:56:31.674865 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:31.674839    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 15:56:34.044180 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.044146    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j"]
May 20 15:56:34.044639 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.044582    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da04dcf0-5363-4b02-b5ba-8e39acd77721" containerName="kube-rbac-proxy"
May 20 15:56:34.044639 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.044600    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="da04dcf0-5363-4b02-b5ba-8e39acd77721" containerName="kube-rbac-proxy"
May 20 15:56:34.044639 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.044615    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbffac89-efb6-4741-b082-16c3d625f6b5" containerName="kube-rbac-proxy"
May 20 15:56:34.044639 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.044624    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbffac89-efb6-4741-b082-16c3d625f6b5" containerName="kube-rbac-proxy"
May 20 15:56:34.044639 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.044640    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbffac89-efb6-4741-b082-16c3d625f6b5" containerName="kserve-container"
May 20 15:56:34.045129 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.044649    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbffac89-efb6-4741-b082-16c3d625f6b5" containerName="kserve-container"
May 20 15:56:34.045129 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.044661    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da04dcf0-5363-4b02-b5ba-8e39acd77721" containerName="kserve-container"
May 20 15:56:34.045129 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.044669    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="da04dcf0-5363-4b02-b5ba-8e39acd77721" containerName="kserve-container"
May 20 15:56:34.045129 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.044682    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13eb4007-0f04-4f0d-8768-7a3de2ab17e4" containerName="ensemble-graph-687f8"
May 20 15:56:34.045129 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.044690    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="13eb4007-0f04-4f0d-8768-7a3de2ab17e4" containerName="ensemble-graph-687f8"
May 20 15:56:34.045129 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.044697    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="763038c5-1d94-4e7e-88bb-a730362c7469" containerName="sequence-graph-4ff76"
May 20 15:56:34.045129 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.044704    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="763038c5-1d94-4e7e-88bb-a730362c7469" containerName="sequence-graph-4ff76"
May 20 15:56:34.045129 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.044790    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="da04dcf0-5363-4b02-b5ba-8e39acd77721" containerName="kube-rbac-proxy"
May 20 15:56:34.045129 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.044803    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="da04dcf0-5363-4b02-b5ba-8e39acd77721" containerName="kserve-container"
May 20 15:56:34.045129 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.044810    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="bbffac89-efb6-4741-b082-16c3d625f6b5" containerName="kube-rbac-proxy"
May 20 15:56:34.045129 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.044816    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="bbffac89-efb6-4741-b082-16c3d625f6b5" containerName="kserve-container"
May 20 15:56:34.045129 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.044823    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="763038c5-1d94-4e7e-88bb-a730362c7469" containerName="sequence-graph-4ff76"
May 20 15:56:34.045129 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.044831    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="13eb4007-0f04-4f0d-8768-7a3de2ab17e4" containerName="ensemble-graph-687f8"
May 20 15:56:34.049087 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.049069    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j"
May 20 15:56:34.051401 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.051375    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-3f928-serving-cert\""
May 20 15:56:34.051401 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.051391    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-3f928-kube-rbac-proxy-sar-config\""
May 20 15:56:34.057137 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.057048    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j"]
May 20 15:56:34.112436 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.112395    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c59413-c05b-43b0-b7d9-69de5d9d8344-openshift-service-ca-bundle\") pod \"ensemble-graph-3f928-6dc74cd58b-jw88j\" (UID: \"93c59413-c05b-43b0-b7d9-69de5d9d8344\") " pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j"
May 20 15:56:34.112624 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.112468    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93c59413-c05b-43b0-b7d9-69de5d9d8344-proxy-tls\") pod \"ensemble-graph-3f928-6dc74cd58b-jw88j\" (UID: \"93c59413-c05b-43b0-b7d9-69de5d9d8344\") " pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j"
May 20 15:56:34.213659 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.213617    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93c59413-c05b-43b0-b7d9-69de5d9d8344-proxy-tls\") pod \"ensemble-graph-3f928-6dc74cd58b-jw88j\" (UID: \"93c59413-c05b-43b0-b7d9-69de5d9d8344\") " pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j"
May 20 15:56:34.213821 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.213704    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c59413-c05b-43b0-b7d9-69de5d9d8344-openshift-service-ca-bundle\") pod \"ensemble-graph-3f928-6dc74cd58b-jw88j\" (UID: \"93c59413-c05b-43b0-b7d9-69de5d9d8344\") " pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j"
May 20 15:56:34.214349 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.214331    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c59413-c05b-43b0-b7d9-69de5d9d8344-openshift-service-ca-bundle\") pod \"ensemble-graph-3f928-6dc74cd58b-jw88j\" (UID: \"93c59413-c05b-43b0-b7d9-69de5d9d8344\") " pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j"
May 20 15:56:34.216228 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.216206    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93c59413-c05b-43b0-b7d9-69de5d9d8344-proxy-tls\") pod \"ensemble-graph-3f928-6dc74cd58b-jw88j\" (UID: \"93c59413-c05b-43b0-b7d9-69de5d9d8344\") " pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j"
May 20 15:56:34.361384 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.361347    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j"
May 20 15:56:34.497476 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.497449    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j"]
May 20 15:56:34.499941 ip-10-0-129-117 kubenswrapper[2568]: W0520 15:56:34.499913    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c59413_c05b_43b0_b7d9_69de5d9d8344.slice/crio-d2896d474c33af1a30fa49e525b5937d1a5117157e2c9f2b09c219ad56454894 WatchSource:0}: Error finding container d2896d474c33af1a30fa49e525b5937d1a5117157e2c9f2b09c219ad56454894: Status 404 returned error can't find the container with id d2896d474c33af1a30fa49e525b5937d1a5117157e2c9f2b09c219ad56454894
May 20 15:56:34.970438 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.970396    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j" event={"ID":"93c59413-c05b-43b0-b7d9-69de5d9d8344","Type":"ContainerStarted","Data":"a803498a268d9640e7a191c7cdafe572b199c59714fa40ee4fc227ed918dc513"}
May 20 15:56:34.970438 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.970436    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j" event={"ID":"93c59413-c05b-43b0-b7d9-69de5d9d8344","Type":"ContainerStarted","Data":"d2896d474c33af1a30fa49e525b5937d1a5117157e2c9f2b09c219ad56454894"}
May 20 15:56:34.970681 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.970539    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j"
May 20 15:56:34.988085 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:34.988037    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j" podStartSLOduration=0.988000963 podStartE2EDuration="988.000963ms" podCreationTimestamp="2026-05-20 15:56:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:56:34.985921946 +0000 UTC m=+1503.924763503" watchObservedRunningTime="2026-05-20 15:56:34.988000963 +0000 UTC m=+1503.926842520"
May 20 15:56:37.774400 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:37.774368    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"
May 20 15:56:37.774764 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:37.774425    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"
May 20 15:56:40.979691 ip-10-0-129-117 kubenswrapper[2568]: I0520 15:56:40.979661    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j"
May 20 16:01:31.701878 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:01:31.701845    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 16:01:31.704138 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:01:31.704115    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 16:04:48.761202 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:48.761162    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j"]
May 20 16:04:48.761941 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:48.761495    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j" podUID="93c59413-c05b-43b0-b7d9-69de5d9d8344" containerName="ensemble-graph-3f928" containerID="cri-o://a803498a268d9640e7a191c7cdafe572b199c59714fa40ee4fc227ed918dc513" gracePeriod=30
May 20 16:04:48.887707 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:48.887674    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"]
May 20 16:04:48.888003 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:48.887976    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c" podUID="022a7db3-b507-4127-9278-3d3308028d02" containerName="kserve-container" containerID="cri-o://5f46792aecb8d4b5d9d2e6fe18dfca4d13ce32163a1e1722cb2ac47f87935801" gracePeriod=30
May 20 16:04:48.888129 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:48.888071    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c" podUID="022a7db3-b507-4127-9278-3d3308028d02" containerName="kube-rbac-proxy" containerID="cri-o://3fbf0499141b51b71bff11f9f5b42c48b6001a3b10e2567e66f15e00c84d1471" gracePeriod=30
May 20 16:04:48.958311 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:48.958275    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"]
May 20 16:04:48.958590 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:48.958568    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn" podUID="0b9ffefe-0aad-4ca0-9b52-59b073671f33" containerName="kserve-container" containerID="cri-o://c19f9a06f96c019063883ef931c4c936200894e7f6d5fc4ceb3a373ad451095a" gracePeriod=30
May 20 16:04:48.958681 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:48.958649    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn" podUID="0b9ffefe-0aad-4ca0-9b52-59b073671f33" containerName="kube-rbac-proxy" containerID="cri-o://ea1db858048e39fe0a046621f042a4b019c06a79ced6b768e81332e2f3bb050c" gracePeriod=30
May 20 16:04:48.971692 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:48.971664    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"]
May 20 16:04:48.975737 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:48.975719    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"
May 20 16:04:48.978490 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:48.977952    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e05eb-predictor-serving-cert\""
May 20 16:04:48.978490 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:48.978304    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e05eb-kube-rbac-proxy-sar-config\""
May 20 16:04:48.994658 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:48.994528    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrq88\" (UniqueName: \"kubernetes.io/projected/f4ac65ba-473b-4af6-bfc2-bce8092a55ce-kube-api-access-wrq88\") pod \"success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq\" (UID: \"f4ac65ba-473b-4af6-bfc2-bce8092a55ce\") " pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"
May 20 16:04:48.995486 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:48.995422    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4ac65ba-473b-4af6-bfc2-bce8092a55ce-proxy-tls\") pod \"success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq\" (UID: \"f4ac65ba-473b-4af6-bfc2-bce8092a55ce\") " pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"
May 20 16:04:48.996208 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:48.996049    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-e05eb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4ac65ba-473b-4af6-bfc2-bce8092a55ce-success-200-isvc-e05eb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq\" (UID: \"f4ac65ba-473b-4af6-bfc2-bce8092a55ce\") " pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"
May 20 16:04:48.999156 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:48.999135    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"]
May 20 16:04:49.036419 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.036336    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"]
May 20 16:04:49.039977 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.039953    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"
May 20 16:04:49.042396 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.042375    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e05eb-kube-rbac-proxy-sar-config\""
May 20 16:04:49.042571 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.042557    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e05eb-predictor-serving-cert\""
May 20 16:04:49.050976 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.050950    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"]
May 20 16:04:49.097125 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.097084    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eda94f7-0a45-42af-ace9-a99be2b1368c-proxy-tls\") pod \"error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj\" (UID: \"0eda94f7-0a45-42af-ace9-a99be2b1368c\") " pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"
May 20 16:04:49.097273 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.097143    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bphhl\" (UniqueName: \"kubernetes.io/projected/0eda94f7-0a45-42af-ace9-a99be2b1368c-kube-api-access-bphhl\") pod \"error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj\" (UID: \"0eda94f7-0a45-42af-ace9-a99be2b1368c\") " pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"
May 20 16:04:49.097273 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.097216    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wrq88\" (UniqueName: \"kubernetes.io/projected/f4ac65ba-473b-4af6-bfc2-bce8092a55ce-kube-api-access-wrq88\") pod \"success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq\" (UID: \"f4ac65ba-473b-4af6-bfc2-bce8092a55ce\") " pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"
May 20 16:04:49.097273 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.097268    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4ac65ba-473b-4af6-bfc2-bce8092a55ce-proxy-tls\") pod \"success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq\" (UID: \"f4ac65ba-473b-4af6-bfc2-bce8092a55ce\") " pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"
May 20 16:04:49.097408 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.097287    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-e05eb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4ac65ba-473b-4af6-bfc2-bce8092a55ce-success-200-isvc-e05eb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq\" (UID: \"f4ac65ba-473b-4af6-bfc2-bce8092a55ce\") " pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"
May 20 16:04:49.097408 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.097359    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-e05eb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0eda94f7-0a45-42af-ace9-a99be2b1368c-error-404-isvc-e05eb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj\" (UID: \"0eda94f7-0a45-42af-ace9-a99be2b1368c\") " pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"
May 20 16:04:49.097408 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:04:49.097393    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-serving-cert: secret "success-200-isvc-e05eb-predictor-serving-cert" not found
May 20 16:04:49.097558 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:04:49.097453    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4ac65ba-473b-4af6-bfc2-bce8092a55ce-proxy-tls podName:f4ac65ba-473b-4af6-bfc2-bce8092a55ce nodeName:}" failed. No retries permitted until 2026-05-20 16:04:49.597432235 +0000 UTC m=+1998.536273776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f4ac65ba-473b-4af6-bfc2-bce8092a55ce-proxy-tls") pod "success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq" (UID: "f4ac65ba-473b-4af6-bfc2-bce8092a55ce") : secret "success-200-isvc-e05eb-predictor-serving-cert" not found
May 20 16:04:49.098058 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.098036    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-e05eb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4ac65ba-473b-4af6-bfc2-bce8092a55ce-success-200-isvc-e05eb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq\" (UID: \"f4ac65ba-473b-4af6-bfc2-bce8092a55ce\") " pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"
May 20 16:04:49.108930 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.108906    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrq88\" (UniqueName: \"kubernetes.io/projected/f4ac65ba-473b-4af6-bfc2-bce8092a55ce-kube-api-access-wrq88\") pod \"success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq\" (UID: \"f4ac65ba-473b-4af6-bfc2-bce8092a55ce\") " pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"
May 20 16:04:49.198085 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.198047    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-e05eb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0eda94f7-0a45-42af-ace9-a99be2b1368c-error-404-isvc-e05eb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj\" (UID: \"0eda94f7-0a45-42af-ace9-a99be2b1368c\") " pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"
May 20 16:04:49.198298 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.198136    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eda94f7-0a45-42af-ace9-a99be2b1368c-proxy-tls\") pod \"error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj\" (UID: \"0eda94f7-0a45-42af-ace9-a99be2b1368c\") " pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"
May 20 16:04:49.198298 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.198168    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bphhl\" (UniqueName: \"kubernetes.io/projected/0eda94f7-0a45-42af-ace9-a99be2b1368c-kube-api-access-bphhl\") pod \"error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj\" (UID: \"0eda94f7-0a45-42af-ace9-a99be2b1368c\") " pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"
May 20 16:04:49.198410 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:04:49.198306    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-serving-cert: secret "error-404-isvc-e05eb-predictor-serving-cert" not found
May 20 16:04:49.198410 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:04:49.198395    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eda94f7-0a45-42af-ace9-a99be2b1368c-proxy-tls podName:0eda94f7-0a45-42af-ace9-a99be2b1368c nodeName:}" failed. No retries permitted until 2026-05-20 16:04:49.698373828 +0000 UTC m=+1998.637215370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0eda94f7-0a45-42af-ace9-a99be2b1368c-proxy-tls") pod "error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj" (UID: "0eda94f7-0a45-42af-ace9-a99be2b1368c") : secret "error-404-isvc-e05eb-predictor-serving-cert" not found
May 20 16:04:49.198759 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.198741    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-e05eb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0eda94f7-0a45-42af-ace9-a99be2b1368c-error-404-isvc-e05eb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj\" (UID: \"0eda94f7-0a45-42af-ace9-a99be2b1368c\") " pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"
May 20 16:04:49.216703 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.216669    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bphhl\" (UniqueName: \"kubernetes.io/projected/0eda94f7-0a45-42af-ace9-a99be2b1368c-kube-api-access-bphhl\") pod \"error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj\" (UID: \"0eda94f7-0a45-42af-ace9-a99be2b1368c\") " pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"
May 20 16:04:49.601111 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.601066    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4ac65ba-473b-4af6-bfc2-bce8092a55ce-proxy-tls\") pod \"success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq\" (UID: \"f4ac65ba-473b-4af6-bfc2-bce8092a55ce\") " pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"
May 20 16:04:49.603659 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.603635    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4ac65ba-473b-4af6-bfc2-bce8092a55ce-proxy-tls\") pod \"success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq\" (UID: \"f4ac65ba-473b-4af6-bfc2-bce8092a55ce\") " pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"
May 20 16:04:49.682384 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.682349    2568 generic.go:358] "Generic (PLEG): container finished" podID="022a7db3-b507-4127-9278-3d3308028d02" containerID="3fbf0499141b51b71bff11f9f5b42c48b6001a3b10e2567e66f15e00c84d1471" exitCode=2
May 20 16:04:49.682590 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.682429    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c" event={"ID":"022a7db3-b507-4127-9278-3d3308028d02","Type":"ContainerDied","Data":"3fbf0499141b51b71bff11f9f5b42c48b6001a3b10e2567e66f15e00c84d1471"}
May 20 16:04:49.684166 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.684142    2568 generic.go:358] "Generic (PLEG): container finished" podID="0b9ffefe-0aad-4ca0-9b52-59b073671f33" containerID="ea1db858048e39fe0a046621f042a4b019c06a79ced6b768e81332e2f3bb050c" exitCode=2
May 20 16:04:49.684283 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.684182    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn" event={"ID":"0b9ffefe-0aad-4ca0-9b52-59b073671f33","Type":"ContainerDied","Data":"ea1db858048e39fe0a046621f042a4b019c06a79ced6b768e81332e2f3bb050c"}
May 20 16:04:49.702304 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.702278    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eda94f7-0a45-42af-ace9-a99be2b1368c-proxy-tls\") pod \"error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj\" (UID: \"0eda94f7-0a45-42af-ace9-a99be2b1368c\") " pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"
May 20 16:04:49.704827 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.704804    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eda94f7-0a45-42af-ace9-a99be2b1368c-proxy-tls\") pod \"error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj\" (UID: \"0eda94f7-0a45-42af-ace9-a99be2b1368c\") " pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"
May 20 16:04:49.888594 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.888489    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"
May 20 16:04:49.953420 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:49.953386    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"
May 20 16:04:50.034402 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:50.034373    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"]
May 20 16:04:50.036673 ip-10-0-129-117 kubenswrapper[2568]: W0520 16:04:50.036646    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ac65ba_473b_4af6_bfc2_bce8092a55ce.slice/crio-28fd16f5335dc13bad45fa5891c58bcc15af13e9f208a796c40d43ddcadba63b WatchSource:0}: Error finding container 28fd16f5335dc13bad45fa5891c58bcc15af13e9f208a796c40d43ddcadba63b: Status 404 returned error can't find the container with id 28fd16f5335dc13bad45fa5891c58bcc15af13e9f208a796c40d43ddcadba63b
May 20 16:04:50.038586 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:50.038570    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 16:04:50.102060 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:50.102034    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"]
May 20 16:04:50.103949 ip-10-0-129-117 kubenswrapper[2568]: W0520 16:04:50.103920    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eda94f7_0a45_42af_ace9_a99be2b1368c.slice/crio-48137aba873bf35a4b6d8f7dbba7dc10957a452111686f0e4ed9a974528ea980 WatchSource:0}: Error finding container 48137aba873bf35a4b6d8f7dbba7dc10957a452111686f0e4ed9a974528ea980: Status 404 returned error can't find the container with id 48137aba873bf35a4b6d8f7dbba7dc10957a452111686f0e4ed9a974528ea980
May 20 16:04:50.689121 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:50.689069    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq" event={"ID":"f4ac65ba-473b-4af6-bfc2-bce8092a55ce","Type":"ContainerStarted","Data":"25596ca59cb42cfa3940ef5718d17b398f535aee0e33689ffd3c1c63260f6d23"}
May 20 16:04:50.689327 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:50.689126    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq" event={"ID":"f4ac65ba-473b-4af6-bfc2-bce8092a55ce","Type":"ContainerStarted","Data":"625ef2131d5c4fbd5b4fc5854accea9d323510c3176893e399f9bfd51bc35e14"}
May 20 16:04:50.689327 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:50.689219    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"
May 20 16:04:50.689327 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:50.689233    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq" event={"ID":"f4ac65ba-473b-4af6-bfc2-bce8092a55ce","Type":"ContainerStarted","Data":"28fd16f5335dc13bad45fa5891c58bcc15af13e9f208a796c40d43ddcadba63b"}
May 20 16:04:50.689327 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:50.689250    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"
May 20 16:04:50.690809 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:50.690781    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj" event={"ID":"0eda94f7-0a45-42af-ace9-a99be2b1368c","Type":"ContainerStarted","Data":"f4c09c1c86924d46f0cba917f71e5dfb87249dff9bd90450fc54a6871f089e58"}
May 20 16:04:50.690938 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:50.690815    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj" event={"ID":"0eda94f7-0a45-42af-ace9-a99be2b1368c","Type":"ContainerStarted","Data":"9b773dcf97ee087e6d326f9b6ddffb7e1c9a0999ec1e5fa3cd4d895504c2ad92"}
May 20 16:04:50.690938 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:50.690827    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj" event={"ID":"0eda94f7-0a45-42af-ace9-a99be2b1368c","Type":"ContainerStarted","Data":"48137aba873bf35a4b6d8f7dbba7dc10957a452111686f0e4ed9a974528ea980"}
May 20 16:04:50.690938 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:50.690913    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"
May 20 16:04:50.691094 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:50.691038    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq" podUID="f4ac65ba-473b-4af6-bfc2-bce8092a55ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 20 16:04:50.707199 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:50.707147    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq" podStartSLOduration=2.707132721 podStartE2EDuration="2.707132721s" podCreationTimestamp="2026-05-20 16:04:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:04:50.706057369 +0000 UTC m=+1999.644898928" watchObservedRunningTime="2026-05-20 16:04:50.707132721 +0000 UTC m=+1999.645974278"
May 20 16:04:50.723567 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:50.723520    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj" podStartSLOduration=1.723506799 podStartE2EDuration="1.723506799s" podCreationTimestamp="2026-05-20 16:04:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:04:50.721253221 +0000 UTC m=+1999.660094778" watchObservedRunningTime="2026-05-20 16:04:50.723506799 +0000 UTC m=+1999.662348356"
May 20 16:04:50.979613 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:50.979521    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j" podUID="93c59413-c05b-43b0-b7d9-69de5d9d8344" containerName="ensemble-graph-3f928" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 16:04:51.694489 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:51.694440    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq" podUID="f4ac65ba-473b-4af6-bfc2-bce8092a55ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 20 16:04:51.694696 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:51.694491    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"
May 20 16:04:51.695053 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:51.695003    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c" podUID="022a7db3-b507-4127-9278-3d3308028d02" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
May 20 16:04:51.695836 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:51.695808    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj" podUID="0eda94f7-0a45-42af-ace9-a99be2b1368c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 20 16:04:51.700209 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:51.700188    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c" podUID="022a7db3-b507-4127-9278-3d3308028d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 20 16:04:52.382810 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.382787    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"
May 20 16:04:52.429153 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.429049    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbrc2\" (UniqueName: \"kubernetes.io/projected/022a7db3-b507-4127-9278-3d3308028d02-kube-api-access-vbrc2\") pod \"022a7db3-b507-4127-9278-3d3308028d02\" (UID: \"022a7db3-b507-4127-9278-3d3308028d02\") "
May 20 16:04:52.429312 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.429156    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/022a7db3-b507-4127-9278-3d3308028d02-proxy-tls\") pod \"022a7db3-b507-4127-9278-3d3308028d02\" (UID: \"022a7db3-b507-4127-9278-3d3308028d02\") "
May 20 16:04:52.429312 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.429212    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-3f928-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/022a7db3-b507-4127-9278-3d3308028d02-success-200-isvc-3f928-kube-rbac-proxy-sar-config\") pod \"022a7db3-b507-4127-9278-3d3308028d02\" (UID: \"022a7db3-b507-4127-9278-3d3308028d02\") "
May 20 16:04:52.429950 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.429733    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/022a7db3-b507-4127-9278-3d3308028d02-success-200-isvc-3f928-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-3f928-kube-rbac-proxy-sar-config") pod "022a7db3-b507-4127-9278-3d3308028d02" (UID: "022a7db3-b507-4127-9278-3d3308028d02"). InnerVolumeSpecName "success-200-isvc-3f928-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:04:52.433910 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.433879    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/022a7db3-b507-4127-9278-3d3308028d02-kube-api-access-vbrc2" (OuterVolumeSpecName: "kube-api-access-vbrc2") pod "022a7db3-b507-4127-9278-3d3308028d02" (UID: "022a7db3-b507-4127-9278-3d3308028d02"). InnerVolumeSpecName "kube-api-access-vbrc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:04:52.435515 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.435486    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/022a7db3-b507-4127-9278-3d3308028d02-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "022a7db3-b507-4127-9278-3d3308028d02" (UID: "022a7db3-b507-4127-9278-3d3308028d02"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:04:52.509547 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.509522    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"
May 20 16:04:52.530365 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.530331    2568 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-3f928-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/022a7db3-b507-4127-9278-3d3308028d02-success-200-isvc-3f928-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:04:52.530365 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.530363    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbrc2\" (UniqueName: \"kubernetes.io/projected/022a7db3-b507-4127-9278-3d3308028d02-kube-api-access-vbrc2\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:04:52.530620 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.530379    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/022a7db3-b507-4127-9278-3d3308028d02-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:04:52.630934 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.630895    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-3f928-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b9ffefe-0aad-4ca0-9b52-59b073671f33-error-404-isvc-3f928-kube-rbac-proxy-sar-config\") pod \"0b9ffefe-0aad-4ca0-9b52-59b073671f33\" (UID: \"0b9ffefe-0aad-4ca0-9b52-59b073671f33\") "
May 20 16:04:52.631141 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.630947    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59kdg\" (UniqueName: \"kubernetes.io/projected/0b9ffefe-0aad-4ca0-9b52-59b073671f33-kube-api-access-59kdg\") pod \"0b9ffefe-0aad-4ca0-9b52-59b073671f33\" (UID: \"0b9ffefe-0aad-4ca0-9b52-59b073671f33\") "
May 20 16:04:52.631141 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.630969    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b9ffefe-0aad-4ca0-9b52-59b073671f33-proxy-tls\") pod \"0b9ffefe-0aad-4ca0-9b52-59b073671f33\" (UID: \"0b9ffefe-0aad-4ca0-9b52-59b073671f33\") "
May 20 16:04:52.631375 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.631345    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b9ffefe-0aad-4ca0-9b52-59b073671f33-error-404-isvc-3f928-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-3f928-kube-rbac-proxy-sar-config") pod "0b9ffefe-0aad-4ca0-9b52-59b073671f33" (UID: "0b9ffefe-0aad-4ca0-9b52-59b073671f33"). InnerVolumeSpecName "error-404-isvc-3f928-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:04:52.633295 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.633268    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b9ffefe-0aad-4ca0-9b52-59b073671f33-kube-api-access-59kdg" (OuterVolumeSpecName: "kube-api-access-59kdg") pod "0b9ffefe-0aad-4ca0-9b52-59b073671f33" (UID: "0b9ffefe-0aad-4ca0-9b52-59b073671f33"). InnerVolumeSpecName "kube-api-access-59kdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:04:52.633295 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.633275    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b9ffefe-0aad-4ca0-9b52-59b073671f33-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0b9ffefe-0aad-4ca0-9b52-59b073671f33" (UID: "0b9ffefe-0aad-4ca0-9b52-59b073671f33"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:04:52.699221 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.699118    2568 generic.go:358] "Generic (PLEG): container finished" podID="0b9ffefe-0aad-4ca0-9b52-59b073671f33" containerID="c19f9a06f96c019063883ef931c4c936200894e7f6d5fc4ceb3a373ad451095a" exitCode=0
May 20 16:04:52.699221 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.699212    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"
May 20 16:04:52.699418 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.699215    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn" event={"ID":"0b9ffefe-0aad-4ca0-9b52-59b073671f33","Type":"ContainerDied","Data":"c19f9a06f96c019063883ef931c4c936200894e7f6d5fc4ceb3a373ad451095a"}
May 20 16:04:52.699418 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.699271    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn" event={"ID":"0b9ffefe-0aad-4ca0-9b52-59b073671f33","Type":"ContainerDied","Data":"59798dc8cb3ef9d84c64f2c446073b398b6a713a7f99505e9a087c71170fe381"}
May 20 16:04:52.699418 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.699296    2568 scope.go:117] "RemoveContainer" containerID="ea1db858048e39fe0a046621f042a4b019c06a79ced6b768e81332e2f3bb050c"
May 20 16:04:52.700738 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.700709    2568 generic.go:358] "Generic (PLEG): container finished" podID="022a7db3-b507-4127-9278-3d3308028d02" containerID="5f46792aecb8d4b5d9d2e6fe18dfca4d13ce32163a1e1722cb2ac47f87935801" exitCode=0
May 20 16:04:52.700876 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.700827    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"
May 20 16:04:52.701000 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.700975    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c" event={"ID":"022a7db3-b507-4127-9278-3d3308028d02","Type":"ContainerDied","Data":"5f46792aecb8d4b5d9d2e6fe18dfca4d13ce32163a1e1722cb2ac47f87935801"}
May 20 16:04:52.701099 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.701044    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c" event={"ID":"022a7db3-b507-4127-9278-3d3308028d02","Type":"ContainerDied","Data":"9a092d017ea03dc37d676bb2e480c142d8225520b99ae88c648ca7093eb423ad"}
May 20 16:04:52.701378 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.701352    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj" podUID="0eda94f7-0a45-42af-ace9-a99be2b1368c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 20 16:04:52.711288 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.711168    2568 scope.go:117] "RemoveContainer" containerID="c19f9a06f96c019063883ef931c4c936200894e7f6d5fc4ceb3a373ad451095a"
May 20 16:04:52.720064 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.720044    2568 scope.go:117] "RemoveContainer" containerID="ea1db858048e39fe0a046621f042a4b019c06a79ced6b768e81332e2f3bb050c"
May 20 16:04:52.720348 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:04:52.720328    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea1db858048e39fe0a046621f042a4b019c06a79ced6b768e81332e2f3bb050c\": container with ID starting with ea1db858048e39fe0a046621f042a4b019c06a79ced6b768e81332e2f3bb050c not found: ID does not exist" containerID="ea1db858048e39fe0a046621f042a4b019c06a79ced6b768e81332e2f3bb050c"
May 20 16:04:52.720425 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.720362    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea1db858048e39fe0a046621f042a4b019c06a79ced6b768e81332e2f3bb050c"} err="failed to get container status \"ea1db858048e39fe0a046621f042a4b019c06a79ced6b768e81332e2f3bb050c\": rpc error: code = NotFound desc = could not find container \"ea1db858048e39fe0a046621f042a4b019c06a79ced6b768e81332e2f3bb050c\": container with ID starting with ea1db858048e39fe0a046621f042a4b019c06a79ced6b768e81332e2f3bb050c not found: ID does not exist"
May 20 16:04:52.720425 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.720387    2568 scope.go:117] "RemoveContainer" containerID="c19f9a06f96c019063883ef931c4c936200894e7f6d5fc4ceb3a373ad451095a"
May 20 16:04:52.720631 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:04:52.720613    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c19f9a06f96c019063883ef931c4c936200894e7f6d5fc4ceb3a373ad451095a\": container with ID starting with c19f9a06f96c019063883ef931c4c936200894e7f6d5fc4ceb3a373ad451095a not found: ID does not exist" containerID="c19f9a06f96c019063883ef931c4c936200894e7f6d5fc4ceb3a373ad451095a"
May 20 16:04:52.720672 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.720638    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c19f9a06f96c019063883ef931c4c936200894e7f6d5fc4ceb3a373ad451095a"} err="failed to get container status \"c19f9a06f96c019063883ef931c4c936200894e7f6d5fc4ceb3a373ad451095a\": rpc error: code = NotFound desc = could not find container \"c19f9a06f96c019063883ef931c4c936200894e7f6d5fc4ceb3a373ad451095a\": container with ID starting with c19f9a06f96c019063883ef931c4c936200894e7f6d5fc4ceb3a373ad451095a not found: ID does not exist"
May 20 16:04:52.720672 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.720655    2568 scope.go:117] "RemoveContainer" containerID="3fbf0499141b51b71bff11f9f5b42c48b6001a3b10e2567e66f15e00c84d1471"
May 20 16:04:52.726962 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.726941    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"]
May 20 16:04:52.729613 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.729594    2568 scope.go:117] "RemoveContainer" containerID="5f46792aecb8d4b5d9d2e6fe18dfca4d13ce32163a1e1722cb2ac47f87935801"
May 20 16:04:52.731142 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.731119    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3f928-predictor-669ffb9b6b-6pg4c"]
May 20 16:04:52.732058 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.732001    2568 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-3f928-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b9ffefe-0aad-4ca0-9b52-59b073671f33-error-404-isvc-3f928-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:04:52.732118 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.732070    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59kdg\" (UniqueName: \"kubernetes.io/projected/0b9ffefe-0aad-4ca0-9b52-59b073671f33-kube-api-access-59kdg\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:04:52.732118 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.732089    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b9ffefe-0aad-4ca0-9b52-59b073671f33-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:04:52.737464 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.737449    2568 scope.go:117] "RemoveContainer" containerID="3fbf0499141b51b71bff11f9f5b42c48b6001a3b10e2567e66f15e00c84d1471"
May 20 16:04:52.737722 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:04:52.737704    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fbf0499141b51b71bff11f9f5b42c48b6001a3b10e2567e66f15e00c84d1471\": container with ID starting with 3fbf0499141b51b71bff11f9f5b42c48b6001a3b10e2567e66f15e00c84d1471 not found: ID does not exist" containerID="3fbf0499141b51b71bff11f9f5b42c48b6001a3b10e2567e66f15e00c84d1471"
May 20 16:04:52.737781 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.737726    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fbf0499141b51b71bff11f9f5b42c48b6001a3b10e2567e66f15e00c84d1471"} err="failed to get container status \"3fbf0499141b51b71bff11f9f5b42c48b6001a3b10e2567e66f15e00c84d1471\": rpc error: code = NotFound desc = could not find container \"3fbf0499141b51b71bff11f9f5b42c48b6001a3b10e2567e66f15e00c84d1471\": container with ID starting with 3fbf0499141b51b71bff11f9f5b42c48b6001a3b10e2567e66f15e00c84d1471 not found: ID does not exist"
May 20 16:04:52.737781 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.737745    2568 scope.go:117] "RemoveContainer" containerID="5f46792aecb8d4b5d9d2e6fe18dfca4d13ce32163a1e1722cb2ac47f87935801"
May 20 16:04:52.737958 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:04:52.737943    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f46792aecb8d4b5d9d2e6fe18dfca4d13ce32163a1e1722cb2ac47f87935801\": container with ID starting with 5f46792aecb8d4b5d9d2e6fe18dfca4d13ce32163a1e1722cb2ac47f87935801 not found: ID does not exist" containerID="5f46792aecb8d4b5d9d2e6fe18dfca4d13ce32163a1e1722cb2ac47f87935801"
May 20 16:04:52.737998 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.737963    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f46792aecb8d4b5d9d2e6fe18dfca4d13ce32163a1e1722cb2ac47f87935801"} err="failed to get container status \"5f46792aecb8d4b5d9d2e6fe18dfca4d13ce32163a1e1722cb2ac47f87935801\": rpc error: code = NotFound desc = could not find container \"5f46792aecb8d4b5d9d2e6fe18dfca4d13ce32163a1e1722cb2ac47f87935801\": container with ID starting with 5f46792aecb8d4b5d9d2e6fe18dfca4d13ce32163a1e1722cb2ac47f87935801 not found: ID does not exist"
May 20 16:04:52.745645 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.745625    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"]
May 20 16:04:52.748036 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:52.748000    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3f928-predictor-787568c877-rvqzn"]
May 20 16:04:53.645110 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:53.645065    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="022a7db3-b507-4127-9278-3d3308028d02" path="/var/lib/kubelet/pods/022a7db3-b507-4127-9278-3d3308028d02/volumes"
May 20 16:04:53.645519 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:53.645504    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b9ffefe-0aad-4ca0-9b52-59b073671f33" path="/var/lib/kubelet/pods/0b9ffefe-0aad-4ca0-9b52-59b073671f33/volumes"
May 20 16:04:55.978001 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:55.977961    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j" podUID="93c59413-c05b-43b0-b7d9-69de5d9d8344" containerName="ensemble-graph-3f928" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 16:04:56.699681 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:56.699647    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"
May 20 16:04:56.700223 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:56.700193    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq" podUID="f4ac65ba-473b-4af6-bfc2-bce8092a55ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 20 16:04:57.705781 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:57.705754    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"
May 20 16:04:57.706443 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:04:57.706415    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj" podUID="0eda94f7-0a45-42af-ace9-a99be2b1368c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 20 16:05:00.977847 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:00.977797    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j" podUID="93c59413-c05b-43b0-b7d9-69de5d9d8344" containerName="ensemble-graph-3f928" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 16:05:00.978282 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:00.977915    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j"
May 20 16:05:04.652010 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.651975    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"]
May 20 16:05:04.652403 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.652278    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x" podUID="0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" containerName="kserve-container" containerID="cri-o://fea7ee8e11233e23eac51f8faea7398b1fd4e87a64c2a8ce0eb2073bbfbd117f" gracePeriod=30
May 20 16:05:04.652403 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.652326    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x" podUID="0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" containerName="kube-rbac-proxy" containerID="cri-o://33b517e82dd506f007822f624f99412b7966b262b738c4ffb7b1a19b03253a6f" gracePeriod=30
May 20 16:05:04.710804 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.710761    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"]
May 20 16:05:04.711219 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.711187    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz" podUID="62f6ac5b-7653-47eb-a82d-c64312fde5ee" containerName="kserve-container" containerID="cri-o://dd44e1460b642e45f1298f5b5dbe27bbee80c484810a0fec246cf50573dee7d8" gracePeriod=30
May 20 16:05:04.711376 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.711237    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz" podUID="62f6ac5b-7653-47eb-a82d-c64312fde5ee" containerName="kube-rbac-proxy" containerID="cri-o://e0b74451a2f48bc7924afbda411451f4fbeee866e29c91f8168e14a5a7e40f26" gracePeriod=30
May 20 16:05:04.736787 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.736756    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"]
May 20 16:05:04.737215 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.737201    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b9ffefe-0aad-4ca0-9b52-59b073671f33" containerName="kube-rbac-proxy"
May 20 16:05:04.737279 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.737218    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b9ffefe-0aad-4ca0-9b52-59b073671f33" containerName="kube-rbac-proxy"
May 20 16:05:04.737279 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.737232    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="022a7db3-b507-4127-9278-3d3308028d02" containerName="kserve-container"
May 20 16:05:04.737279 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.737238    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="022a7db3-b507-4127-9278-3d3308028d02" containerName="kserve-container"
May 20 16:05:04.737279 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.737258    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b9ffefe-0aad-4ca0-9b52-59b073671f33" containerName="kserve-container"
May 20 16:05:04.737279 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.737265    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b9ffefe-0aad-4ca0-9b52-59b073671f33" containerName="kserve-container"
May 20 16:05:04.737279 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.737272    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="022a7db3-b507-4127-9278-3d3308028d02" containerName="kube-rbac-proxy"
May 20 16:05:04.737279 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.737277    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="022a7db3-b507-4127-9278-3d3308028d02" containerName="kube-rbac-proxy"
May 20 16:05:04.737495 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.737338    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="022a7db3-b507-4127-9278-3d3308028d02" containerName="kube-rbac-proxy"
May 20 16:05:04.737495 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.737348    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="0b9ffefe-0aad-4ca0-9b52-59b073671f33" containerName="kube-rbac-proxy"
May 20 16:05:04.737495 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.737358    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="022a7db3-b507-4127-9278-3d3308028d02" containerName="kserve-container"
May 20 16:05:04.737495 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.737365    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="0b9ffefe-0aad-4ca0-9b52-59b073671f33" containerName="kserve-container"
May 20 16:05:04.739710 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.739683    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"
May 20 16:05:04.742435 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.742411    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fd7d8-predictor-serving-cert\""
May 20 16:05:04.742539 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.742417    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fd7d8-kube-rbac-proxy-sar-config\""
May 20 16:05:04.750960 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.750932    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"]
May 20 16:05:04.836374 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.836342    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"]
May 20 16:05:04.839030 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.838994    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"
May 20 16:05:04.840847 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.840821    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/722a5aa0-74ba-479e-91d4-e5db4b6b91e4-proxy-tls\") pod \"success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56\" (UID: \"722a5aa0-74ba-479e-91d4-e5db4b6b91e4\") " pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"
May 20 16:05:04.840967 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.840864    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtc58\" (UniqueName: \"kubernetes.io/projected/722a5aa0-74ba-479e-91d4-e5db4b6b91e4-kube-api-access-qtc58\") pod \"success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56\" (UID: \"722a5aa0-74ba-479e-91d4-e5db4b6b91e4\") " pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"
May 20 16:05:04.841037 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.840980    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-fd7d8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/722a5aa0-74ba-479e-91d4-e5db4b6b91e4-success-200-isvc-fd7d8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56\" (UID: \"722a5aa0-74ba-479e-91d4-e5db4b6b91e4\") " pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"
May 20 16:05:04.841256 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.841240    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fd7d8-predictor-serving-cert\""
May 20 16:05:04.841516 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.841497    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fd7d8-kube-rbac-proxy-sar-config\""
May 20 16:05:04.851970 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.851863    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"]
May 20 16:05:04.941742 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.941648    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-fd7d8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/722a5aa0-74ba-479e-91d4-e5db4b6b91e4-success-200-isvc-fd7d8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56\" (UID: \"722a5aa0-74ba-479e-91d4-e5db4b6b91e4\") " pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"
May 20 16:05:04.941742 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.941705    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/722a5aa0-74ba-479e-91d4-e5db4b6b91e4-proxy-tls\") pod \"success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56\" (UID: \"722a5aa0-74ba-479e-91d4-e5db4b6b91e4\") " pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"
May 20 16:05:04.941742 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.941742    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-fd7d8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84084650-0c60-412c-af89-602e1b8a8101-error-404-isvc-fd7d8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24\" (UID: \"84084650-0c60-412c-af89-602e1b8a8101\") " pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"
May 20 16:05:04.942046 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.941765    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qtc58\" (UniqueName: \"kubernetes.io/projected/722a5aa0-74ba-479e-91d4-e5db4b6b91e4-kube-api-access-qtc58\") pod \"success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56\" (UID: \"722a5aa0-74ba-479e-91d4-e5db4b6b91e4\") " pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"
May 20 16:05:04.942046 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.941806    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84084650-0c60-412c-af89-602e1b8a8101-proxy-tls\") pod \"error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24\" (UID: \"84084650-0c60-412c-af89-602e1b8a8101\") " pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"
May 20 16:05:04.942046 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:05:04.941888    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-serving-cert: secret "success-200-isvc-fd7d8-predictor-serving-cert" not found
May 20 16:05:04.942046 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:05:04.941953    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/722a5aa0-74ba-479e-91d4-e5db4b6b91e4-proxy-tls podName:722a5aa0-74ba-479e-91d4-e5db4b6b91e4 nodeName:}" failed. No retries permitted until 2026-05-20 16:05:05.441934631 +0000 UTC m=+2014.380776168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/722a5aa0-74ba-479e-91d4-e5db4b6b91e4-proxy-tls") pod "success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56" (UID: "722a5aa0-74ba-479e-91d4-e5db4b6b91e4") : secret "success-200-isvc-fd7d8-predictor-serving-cert" not found
May 20 16:05:04.942046 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.941942    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvwxk\" (UniqueName: \"kubernetes.io/projected/84084650-0c60-412c-af89-602e1b8a8101-kube-api-access-zvwxk\") pod \"error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24\" (UID: \"84084650-0c60-412c-af89-602e1b8a8101\") " pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"
May 20 16:05:04.942411 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.942392    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-fd7d8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/722a5aa0-74ba-479e-91d4-e5db4b6b91e4-success-200-isvc-fd7d8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56\" (UID: \"722a5aa0-74ba-479e-91d4-e5db4b6b91e4\") " pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"
May 20 16:05:04.950672 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:04.950646    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtc58\" (UniqueName: \"kubernetes.io/projected/722a5aa0-74ba-479e-91d4-e5db4b6b91e4-kube-api-access-qtc58\") pod \"success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56\" (UID: \"722a5aa0-74ba-479e-91d4-e5db4b6b91e4\") " pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"
May 20 16:05:05.042986 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:05.042944    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84084650-0c60-412c-af89-602e1b8a8101-proxy-tls\") pod \"error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24\" (UID: \"84084650-0c60-412c-af89-602e1b8a8101\") " pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"
May 20 16:05:05.042986 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:05.042985    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zvwxk\" (UniqueName: \"kubernetes.io/projected/84084650-0c60-412c-af89-602e1b8a8101-kube-api-access-zvwxk\") pod \"error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24\" (UID: \"84084650-0c60-412c-af89-602e1b8a8101\") " pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"
May 20 16:05:05.043277 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:05.043133    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-fd7d8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84084650-0c60-412c-af89-602e1b8a8101-error-404-isvc-fd7d8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24\" (UID: \"84084650-0c60-412c-af89-602e1b8a8101\") " pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"
May 20 16:05:05.043724 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:05.043693    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-fd7d8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84084650-0c60-412c-af89-602e1b8a8101-error-404-isvc-fd7d8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24\" (UID: \"84084650-0c60-412c-af89-602e1b8a8101\") " pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"
May 20 16:05:05.045674 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:05.045650    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84084650-0c60-412c-af89-602e1b8a8101-proxy-tls\") pod \"error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24\" (UID: \"84084650-0c60-412c-af89-602e1b8a8101\") " pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"
May 20 16:05:05.052043 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:05.052004    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvwxk\" (UniqueName: \"kubernetes.io/projected/84084650-0c60-412c-af89-602e1b8a8101-kube-api-access-zvwxk\") pod \"error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24\" (UID: \"84084650-0c60-412c-af89-602e1b8a8101\") " pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"
May 20 16:05:05.154816 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:05.154766    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"
May 20 16:05:05.295358 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:05.295333    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"]
May 20 16:05:05.297537 ip-10-0-129-117 kubenswrapper[2568]: W0520 16:05:05.297508    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84084650_0c60_412c_af89_602e1b8a8101.slice/crio-cab6b50dc29305cc42b1ddaa18708f19baa1bd1f12719b03b5f29ee6f8d89370 WatchSource:0}: Error finding container cab6b50dc29305cc42b1ddaa18708f19baa1bd1f12719b03b5f29ee6f8d89370: Status 404 returned error can't find the container with id cab6b50dc29305cc42b1ddaa18708f19baa1bd1f12719b03b5f29ee6f8d89370
May 20 16:05:05.446574 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:05.446541    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/722a5aa0-74ba-479e-91d4-e5db4b6b91e4-proxy-tls\") pod \"success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56\" (UID: \"722a5aa0-74ba-479e-91d4-e5db4b6b91e4\") " pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"
May 20 16:05:05.448950 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:05.448931    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/722a5aa0-74ba-479e-91d4-e5db4b6b91e4-proxy-tls\") pod \"success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56\" (UID: \"722a5aa0-74ba-479e-91d4-e5db4b6b91e4\") " pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"
May 20 16:05:05.652974 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:05.652931    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"
May 20 16:05:05.752566 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:05.752496    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24" event={"ID":"84084650-0c60-412c-af89-602e1b8a8101","Type":"ContainerStarted","Data":"9679969d6747a4ab2bce30dcd8ed1a01a3bc7167f684a9ece4539c5e462a72f8"}
May 20 16:05:05.752566 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:05.752540    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24" event={"ID":"84084650-0c60-412c-af89-602e1b8a8101","Type":"ContainerStarted","Data":"859cd27f334a62a8392894775dd886b43e3b9ee7efb9513358d9adc9fd683890"}
May 20 16:05:05.752566 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:05.752553    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24" event={"ID":"84084650-0c60-412c-af89-602e1b8a8101","Type":"ContainerStarted","Data":"cab6b50dc29305cc42b1ddaa18708f19baa1bd1f12719b03b5f29ee6f8d89370"}
May 20 16:05:05.752827 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:05.752647    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"
May 20 16:05:05.754359 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:05.754331    2568 generic.go:358] "Generic (PLEG): container finished" podID="0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" containerID="33b517e82dd506f007822f624f99412b7966b262b738c4ffb7b1a19b03253a6f" exitCode=2
May 20 16:05:05.754488 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:05.754398    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x" event={"ID":"0a8d08a2-7d2b-4b3f-9064-8c59b18cab79","Type":"ContainerDied","Data":"33b517e82dd506f007822f624f99412b7966b262b738c4ffb7b1a19b03253a6f"}
May 20 16:05:05.756154 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:05.756129    2568 generic.go:358] "Generic (PLEG): container finished" podID="62f6ac5b-7653-47eb-a82d-c64312fde5ee" containerID="e0b74451a2f48bc7924afbda411451f4fbeee866e29c91f8168e14a5a7e40f26" exitCode=2
May 20 16:05:05.756272 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:05.756200    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz" event={"ID":"62f6ac5b-7653-47eb-a82d-c64312fde5ee","Type":"ContainerDied","Data":"e0b74451a2f48bc7924afbda411451f4fbeee866e29c91f8168e14a5a7e40f26"}
May 20 16:05:05.775172 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:05.774977    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24" podStartSLOduration=1.774955804 podStartE2EDuration="1.774955804s" podCreationTimestamp="2026-05-20 16:05:04 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:05:05.770919943 +0000 UTC m=+2014.709761500" watchObservedRunningTime="2026-05-20 16:05:05.774955804 +0000 UTC m=+2014.713797363"
May 20 16:05:05.798603 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:05.798578    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"]
May 20 16:05:05.800761 ip-10-0-129-117 kubenswrapper[2568]: W0520 16:05:05.800733    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722a5aa0_74ba_479e_91d4_e5db4b6b91e4.slice/crio-645550bfaf1ea41100a9af8c1a644ffaafe504a62abc6b950607e85782d2752c WatchSource:0}: Error finding container 645550bfaf1ea41100a9af8c1a644ffaafe504a62abc6b950607e85782d2752c: Status 404 returned error can't find the container with id 645550bfaf1ea41100a9af8c1a644ffaafe504a62abc6b950607e85782d2752c
May 20 16:05:05.979703 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:05.979663    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j" podUID="93c59413-c05b-43b0-b7d9-69de5d9d8344" containerName="ensemble-graph-3f928" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 16:05:06.700267 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:06.700220    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq" podUID="f4ac65ba-473b-4af6-bfc2-bce8092a55ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 20 16:05:06.761297 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:06.761257    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56" event={"ID":"722a5aa0-74ba-479e-91d4-e5db4b6b91e4","Type":"ContainerStarted","Data":"ee01c263670252341de6fa275890ae9fd5b017badf97090fe6879b0c77d7fa06"}
May 20 16:05:06.761480 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:06.761305    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56" event={"ID":"722a5aa0-74ba-479e-91d4-e5db4b6b91e4","Type":"ContainerStarted","Data":"4d3d68a20fc9274367ea66fb3f9a8a2148f849f030fd63743e569e10f1197ed2"}
May 20 16:05:06.761480 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:06.761320    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56" event={"ID":"722a5aa0-74ba-479e-91d4-e5db4b6b91e4","Type":"ContainerStarted","Data":"645550bfaf1ea41100a9af8c1a644ffaafe504a62abc6b950607e85782d2752c"}
May 20 16:05:06.761578 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:06.761484    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"
May 20 16:05:06.761627 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:06.761601    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"
May 20 16:05:06.762924 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:06.762900    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24" podUID="84084650-0c60-412c-af89-602e1b8a8101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 20 16:05:06.785591 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:06.785536    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56" podStartSLOduration=2.785520553 podStartE2EDuration="2.785520553s" podCreationTimestamp="2026-05-20 16:05:04 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:05:06.782357868 +0000 UTC m=+2015.721199425" watchObservedRunningTime="2026-05-20 16:05:06.785520553 +0000 UTC m=+2015.724362109"
May 20 16:05:07.706541 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:07.706504    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj" podUID="0eda94f7-0a45-42af-ace9-a99be2b1368c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 20 16:05:07.765097 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:07.765043    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24" podUID="84084650-0c60-412c-af89-602e1b8a8101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 20 16:05:07.765268 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:07.765149    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"
May 20 16:05:07.766141 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:07.766118    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56" podUID="722a5aa0-74ba-479e-91d4-e5db4b6b91e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 20 16:05:07.767437 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:07.767414    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz" podUID="62f6ac5b-7653-47eb-a82d-c64312fde5ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
May 20 16:05:07.767502 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:07.767420    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x" podUID="0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
May 20 16:05:07.773665 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:07.773612    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz" podUID="62f6ac5b-7653-47eb-a82d-c64312fde5ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
May 20 16:05:07.773928 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:07.773905    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x" podUID="0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 20 16:05:08.418276 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.418250    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"
May 20 16:05:08.476474 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.476367    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-80b6e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a8d08a2-7d2b-4b3f-9064-8c59b18cab79-error-404-isvc-80b6e-kube-rbac-proxy-sar-config\") pod \"0a8d08a2-7d2b-4b3f-9064-8c59b18cab79\" (UID: \"0a8d08a2-7d2b-4b3f-9064-8c59b18cab79\") "
May 20 16:05:08.476653 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.476526    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjhnp\" (UniqueName: \"kubernetes.io/projected/0a8d08a2-7d2b-4b3f-9064-8c59b18cab79-kube-api-access-xjhnp\") pod \"0a8d08a2-7d2b-4b3f-9064-8c59b18cab79\" (UID: \"0a8d08a2-7d2b-4b3f-9064-8c59b18cab79\") "
May 20 16:05:08.476879 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.476849    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8d08a2-7d2b-4b3f-9064-8c59b18cab79-error-404-isvc-80b6e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-80b6e-kube-rbac-proxy-sar-config") pod "0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" (UID: "0a8d08a2-7d2b-4b3f-9064-8c59b18cab79"). InnerVolumeSpecName "error-404-isvc-80b6e-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:05:08.476990 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.476972    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a8d08a2-7d2b-4b3f-9064-8c59b18cab79-proxy-tls\") pod \"0a8d08a2-7d2b-4b3f-9064-8c59b18cab79\" (UID: \"0a8d08a2-7d2b-4b3f-9064-8c59b18cab79\") "
May 20 16:05:08.477378 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.477357    2568 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-80b6e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a8d08a2-7d2b-4b3f-9064-8c59b18cab79-error-404-isvc-80b6e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:05:08.479370 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.479342    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8d08a2-7d2b-4b3f-9064-8c59b18cab79-kube-api-access-xjhnp" (OuterVolumeSpecName: "kube-api-access-xjhnp") pod "0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" (UID: "0a8d08a2-7d2b-4b3f-9064-8c59b18cab79"). InnerVolumeSpecName "kube-api-access-xjhnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:05:08.479492 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.479412    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a8d08a2-7d2b-4b3f-9064-8c59b18cab79-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" (UID: "0a8d08a2-7d2b-4b3f-9064-8c59b18cab79"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:05:08.578451 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.578401    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjhnp\" (UniqueName: \"kubernetes.io/projected/0a8d08a2-7d2b-4b3f-9064-8c59b18cab79-kube-api-access-xjhnp\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:05:08.578451 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.578430    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a8d08a2-7d2b-4b3f-9064-8c59b18cab79-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:05:08.745541 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.745512    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"
May 20 16:05:08.775959 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.775911    2568 generic.go:358] "Generic (PLEG): container finished" podID="62f6ac5b-7653-47eb-a82d-c64312fde5ee" containerID="dd44e1460b642e45f1298f5b5dbe27bbee80c484810a0fec246cf50573dee7d8" exitCode=0
May 20 16:05:08.776188 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.776161    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz" event={"ID":"62f6ac5b-7653-47eb-a82d-c64312fde5ee","Type":"ContainerDied","Data":"dd44e1460b642e45f1298f5b5dbe27bbee80c484810a0fec246cf50573dee7d8"}
May 20 16:05:08.776283 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.776205    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz" event={"ID":"62f6ac5b-7653-47eb-a82d-c64312fde5ee","Type":"ContainerDied","Data":"fd384be36fd99b519b62e21f8fa7155f49e86a385b1e745229e961cdd9814edc"}
May 20 16:05:08.776283 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.776227    2568 scope.go:117] "RemoveContainer" containerID="e0b74451a2f48bc7924afbda411451f4fbeee866e29c91f8168e14a5a7e40f26"
May 20 16:05:08.776788 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.776386    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"
May 20 16:05:08.779594 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.779563    2568 generic.go:358] "Generic (PLEG): container finished" podID="0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" containerID="fea7ee8e11233e23eac51f8faea7398b1fd4e87a64c2a8ce0eb2073bbfbd117f" exitCode=0
May 20 16:05:08.779853 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.779821    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x" event={"ID":"0a8d08a2-7d2b-4b3f-9064-8c59b18cab79","Type":"ContainerDied","Data":"fea7ee8e11233e23eac51f8faea7398b1fd4e87a64c2a8ce0eb2073bbfbd117f"}
May 20 16:05:08.779993 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.779867    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x" event={"ID":"0a8d08a2-7d2b-4b3f-9064-8c59b18cab79","Type":"ContainerDied","Data":"f35da4033b944b1952c5f1369123a74559787a0a535d0fe9d3aab785da440fea"}
May 20 16:05:08.779993 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.779964    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"
May 20 16:05:08.780266 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.780214    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56" podUID="722a5aa0-74ba-479e-91d4-e5db4b6b91e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 20 16:05:08.786859 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.786839    2568 scope.go:117] "RemoveContainer" containerID="dd44e1460b642e45f1298f5b5dbe27bbee80c484810a0fec246cf50573dee7d8"
May 20 16:05:08.796465 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.796443    2568 scope.go:117] "RemoveContainer" containerID="e0b74451a2f48bc7924afbda411451f4fbeee866e29c91f8168e14a5a7e40f26"
May 20 16:05:08.796719 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:05:08.796699    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0b74451a2f48bc7924afbda411451f4fbeee866e29c91f8168e14a5a7e40f26\": container with ID starting with e0b74451a2f48bc7924afbda411451f4fbeee866e29c91f8168e14a5a7e40f26 not found: ID does not exist" containerID="e0b74451a2f48bc7924afbda411451f4fbeee866e29c91f8168e14a5a7e40f26"
May 20 16:05:08.796777 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.796728    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0b74451a2f48bc7924afbda411451f4fbeee866e29c91f8168e14a5a7e40f26"} err="failed to get container status \"e0b74451a2f48bc7924afbda411451f4fbeee866e29c91f8168e14a5a7e40f26\": rpc error: code = NotFound desc = could not find container \"e0b74451a2f48bc7924afbda411451f4fbeee866e29c91f8168e14a5a7e40f26\": container with ID starting with e0b74451a2f48bc7924afbda411451f4fbeee866e29c91f8168e14a5a7e40f26 not found: ID does not exist"
May 20 16:05:08.796777 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.796754    2568 scope.go:117] "RemoveContainer" containerID="dd44e1460b642e45f1298f5b5dbe27bbee80c484810a0fec246cf50573dee7d8"
May 20 16:05:08.796961 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:05:08.796938    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd44e1460b642e45f1298f5b5dbe27bbee80c484810a0fec246cf50573dee7d8\": container with ID starting with dd44e1460b642e45f1298f5b5dbe27bbee80c484810a0fec246cf50573dee7d8 not found: ID does not exist" containerID="dd44e1460b642e45f1298f5b5dbe27bbee80c484810a0fec246cf50573dee7d8"
May 20 16:05:08.796999 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.796965    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd44e1460b642e45f1298f5b5dbe27bbee80c484810a0fec246cf50573dee7d8"} err="failed to get container status \"dd44e1460b642e45f1298f5b5dbe27bbee80c484810a0fec246cf50573dee7d8\": rpc error: code = NotFound desc = could not find container \"dd44e1460b642e45f1298f5b5dbe27bbee80c484810a0fec246cf50573dee7d8\": container with ID starting with dd44e1460b642e45f1298f5b5dbe27bbee80c484810a0fec246cf50573dee7d8 not found: ID does not exist"
May 20 16:05:08.796999 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.796983    2568 scope.go:117] "RemoveContainer" containerID="33b517e82dd506f007822f624f99412b7966b262b738c4ffb7b1a19b03253a6f"
May 20 16:05:08.804684 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.804664    2568 scope.go:117] "RemoveContainer" containerID="fea7ee8e11233e23eac51f8faea7398b1fd4e87a64c2a8ce0eb2073bbfbd117f"
May 20 16:05:08.808049 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.808009    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"]
May 20 16:05:08.813950 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.813926    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-80b6e-predictor-7f59bbd875-8td9x"]
May 20 16:05:08.814244 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.814225    2568 scope.go:117] "RemoveContainer" containerID="33b517e82dd506f007822f624f99412b7966b262b738c4ffb7b1a19b03253a6f"
May 20 16:05:08.814538 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:05:08.814518    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33b517e82dd506f007822f624f99412b7966b262b738c4ffb7b1a19b03253a6f\": container with ID starting with 33b517e82dd506f007822f624f99412b7966b262b738c4ffb7b1a19b03253a6f not found: ID does not exist" containerID="33b517e82dd506f007822f624f99412b7966b262b738c4ffb7b1a19b03253a6f"
May 20 16:05:08.814596 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.814545    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33b517e82dd506f007822f624f99412b7966b262b738c4ffb7b1a19b03253a6f"} err="failed to get container status \"33b517e82dd506f007822f624f99412b7966b262b738c4ffb7b1a19b03253a6f\": rpc error: code = NotFound desc = could not find container \"33b517e82dd506f007822f624f99412b7966b262b738c4ffb7b1a19b03253a6f\": container with ID starting with 33b517e82dd506f007822f624f99412b7966b262b738c4ffb7b1a19b03253a6f not found: ID does not exist"
May 20 16:05:08.814596 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.814565    2568 scope.go:117] "RemoveContainer" containerID="fea7ee8e11233e23eac51f8faea7398b1fd4e87a64c2a8ce0eb2073bbfbd117f"
May 20 16:05:08.814822 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:05:08.814802    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fea7ee8e11233e23eac51f8faea7398b1fd4e87a64c2a8ce0eb2073bbfbd117f\": container with ID starting with fea7ee8e11233e23eac51f8faea7398b1fd4e87a64c2a8ce0eb2073bbfbd117f not found: ID does not exist" containerID="fea7ee8e11233e23eac51f8faea7398b1fd4e87a64c2a8ce0eb2073bbfbd117f"
May 20 16:05:08.814891 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.814830    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fea7ee8e11233e23eac51f8faea7398b1fd4e87a64c2a8ce0eb2073bbfbd117f"} err="failed to get container status \"fea7ee8e11233e23eac51f8faea7398b1fd4e87a64c2a8ce0eb2073bbfbd117f\": rpc error: code = NotFound desc = could not find container \"fea7ee8e11233e23eac51f8faea7398b1fd4e87a64c2a8ce0eb2073bbfbd117f\": container with ID starting with fea7ee8e11233e23eac51f8faea7398b1fd4e87a64c2a8ce0eb2073bbfbd117f not found: ID does not exist"
May 20 16:05:08.880315 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.880278    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62f6ac5b-7653-47eb-a82d-c64312fde5ee-proxy-tls\") pod \"62f6ac5b-7653-47eb-a82d-c64312fde5ee\" (UID: \"62f6ac5b-7653-47eb-a82d-c64312fde5ee\") "
May 20 16:05:08.880504 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.880415    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-80b6e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62f6ac5b-7653-47eb-a82d-c64312fde5ee-success-200-isvc-80b6e-kube-rbac-proxy-sar-config\") pod \"62f6ac5b-7653-47eb-a82d-c64312fde5ee\" (UID: \"62f6ac5b-7653-47eb-a82d-c64312fde5ee\") "
May 20 16:05:08.880504 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.880439    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6snf5\" (UniqueName: \"kubernetes.io/projected/62f6ac5b-7653-47eb-a82d-c64312fde5ee-kube-api-access-6snf5\") pod \"62f6ac5b-7653-47eb-a82d-c64312fde5ee\" (UID: \"62f6ac5b-7653-47eb-a82d-c64312fde5ee\") "
May 20 16:05:08.882101 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.881011    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f6ac5b-7653-47eb-a82d-c64312fde5ee-success-200-isvc-80b6e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-80b6e-kube-rbac-proxy-sar-config") pod "62f6ac5b-7653-47eb-a82d-c64312fde5ee" (UID: "62f6ac5b-7653-47eb-a82d-c64312fde5ee"). InnerVolumeSpecName "success-200-isvc-80b6e-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:05:08.883514 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.883477    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f6ac5b-7653-47eb-a82d-c64312fde5ee-kube-api-access-6snf5" (OuterVolumeSpecName: "kube-api-access-6snf5") pod "62f6ac5b-7653-47eb-a82d-c64312fde5ee" (UID: "62f6ac5b-7653-47eb-a82d-c64312fde5ee"). InnerVolumeSpecName "kube-api-access-6snf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:05:08.888144 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.888117    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f6ac5b-7653-47eb-a82d-c64312fde5ee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "62f6ac5b-7653-47eb-a82d-c64312fde5ee" (UID: "62f6ac5b-7653-47eb-a82d-c64312fde5ee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:05:08.981546 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.981507    2568 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-80b6e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62f6ac5b-7653-47eb-a82d-c64312fde5ee-success-200-isvc-80b6e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:05:08.981546 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.981542    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6snf5\" (UniqueName: \"kubernetes.io/projected/62f6ac5b-7653-47eb-a82d-c64312fde5ee-kube-api-access-6snf5\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:05:08.981546 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:08.981555    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62f6ac5b-7653-47eb-a82d-c64312fde5ee-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:05:09.098594 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:09.098556    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"]
May 20 16:05:09.102727 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:09.102702    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-80b6e-predictor-789b9765b-ldvnz"]
May 20 16:05:09.644265 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:09.644226    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" path="/var/lib/kubelet/pods/0a8d08a2-7d2b-4b3f-9064-8c59b18cab79/volumes"
May 20 16:05:09.644772 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:09.644751    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62f6ac5b-7653-47eb-a82d-c64312fde5ee" path="/var/lib/kubelet/pods/62f6ac5b-7653-47eb-a82d-c64312fde5ee/volumes"
May 20 16:05:10.979413 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:10.979371    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j" podUID="93c59413-c05b-43b0-b7d9-69de5d9d8344" containerName="ensemble-graph-3f928" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 16:05:12.772208 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:12.772168    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"
May 20 16:05:12.772916 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:12.772878    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24" podUID="84084650-0c60-412c-af89-602e1b8a8101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 20 16:05:13.785959 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:13.785914    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"
May 20 16:05:13.786635 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:13.786608    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56" podUID="722a5aa0-74ba-479e-91d4-e5db4b6b91e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 20 16:05:15.979306 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:15.979257    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j" podUID="93c59413-c05b-43b0-b7d9-69de5d9d8344" containerName="ensemble-graph-3f928" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 16:05:16.700567 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:16.700529    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq" podUID="f4ac65ba-473b-4af6-bfc2-bce8092a55ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 20 16:05:17.706757 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:17.706721    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj" podUID="0eda94f7-0a45-42af-ace9-a99be2b1368c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 20 16:05:18.818296 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:18.818265    2568 generic.go:358] "Generic (PLEG): container finished" podID="93c59413-c05b-43b0-b7d9-69de5d9d8344" containerID="a803498a268d9640e7a191c7cdafe572b199c59714fa40ee4fc227ed918dc513" exitCode=0
May 20 16:05:18.818661 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:18.818335    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j" event={"ID":"93c59413-c05b-43b0-b7d9-69de5d9d8344","Type":"ContainerDied","Data":"a803498a268d9640e7a191c7cdafe572b199c59714fa40ee4fc227ed918dc513"}
May 20 16:05:18.921417 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:18.921395    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j"
May 20 16:05:19.070116 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:19.070081    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c59413-c05b-43b0-b7d9-69de5d9d8344-openshift-service-ca-bundle\") pod \"93c59413-c05b-43b0-b7d9-69de5d9d8344\" (UID: \"93c59413-c05b-43b0-b7d9-69de5d9d8344\") "
May 20 16:05:19.070116 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:19.070122    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93c59413-c05b-43b0-b7d9-69de5d9d8344-proxy-tls\") pod \"93c59413-c05b-43b0-b7d9-69de5d9d8344\" (UID: \"93c59413-c05b-43b0-b7d9-69de5d9d8344\") "
May 20 16:05:19.070504 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:19.070483    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c59413-c05b-43b0-b7d9-69de5d9d8344-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "93c59413-c05b-43b0-b7d9-69de5d9d8344" (UID: "93c59413-c05b-43b0-b7d9-69de5d9d8344"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:05:19.072554 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:19.072514    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c59413-c05b-43b0-b7d9-69de5d9d8344-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "93c59413-c05b-43b0-b7d9-69de5d9d8344" (UID: "93c59413-c05b-43b0-b7d9-69de5d9d8344"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:05:19.170895 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:19.170852    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c59413-c05b-43b0-b7d9-69de5d9d8344-openshift-service-ca-bundle\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:05:19.170895 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:19.170899    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93c59413-c05b-43b0-b7d9-69de5d9d8344-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:05:19.823401 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:19.823376    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j"
May 20 16:05:19.823859 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:19.823379    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j" event={"ID":"93c59413-c05b-43b0-b7d9-69de5d9d8344","Type":"ContainerDied","Data":"d2896d474c33af1a30fa49e525b5937d1a5117157e2c9f2b09c219ad56454894"}
May 20 16:05:19.823859 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:19.823491    2568 scope.go:117] "RemoveContainer" containerID="a803498a268d9640e7a191c7cdafe572b199c59714fa40ee4fc227ed918dc513"
May 20 16:05:19.838757 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:19.838726    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j"]
May 20 16:05:19.841164 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:19.841141    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-3f928-6dc74cd58b-jw88j"]
May 20 16:05:21.645331 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:21.645299    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93c59413-c05b-43b0-b7d9-69de5d9d8344" path="/var/lib/kubelet/pods/93c59413-c05b-43b0-b7d9-69de5d9d8344/volumes"
May 20 16:05:22.772846 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:22.772807    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24" podUID="84084650-0c60-412c-af89-602e1b8a8101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 20 16:05:23.787073 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:23.787000    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56" podUID="722a5aa0-74ba-479e-91d4-e5db4b6b91e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 20 16:05:26.700227 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:26.700166    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq" podUID="f4ac65ba-473b-4af6-bfc2-bce8092a55ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 20 16:05:27.707271 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:27.707230    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj" podUID="0eda94f7-0a45-42af-ace9-a99be2b1368c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 20 16:05:32.772815 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:32.772764    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24" podUID="84084650-0c60-412c-af89-602e1b8a8101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 20 16:05:33.787177 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:33.787135    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56" podUID="722a5aa0-74ba-479e-91d4-e5db4b6b91e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 20 16:05:36.700748 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:36.700705    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq" podUID="f4ac65ba-473b-4af6-bfc2-bce8092a55ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 20 16:05:37.707884 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:37.707852    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"
May 20 16:05:42.773139 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:42.773069    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24" podUID="84084650-0c60-412c-af89-602e1b8a8101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 20 16:05:43.787476 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:43.787432    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56" podUID="722a5aa0-74ba-479e-91d4-e5db4b6b91e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 20 16:05:46.701806 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:46.701771    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"
May 20 16:05:52.773360 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:52.773252    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24" podUID="84084650-0c60-412c-af89-602e1b8a8101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 20 16:05:53.786910 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:05:53.786873    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56" podUID="722a5aa0-74ba-479e-91d4-e5db4b6b91e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 20 16:06:02.773933 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:02.773903    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"
May 20 16:06:03.787999 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:03.787965    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"
May 20 16:06:09.270150 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.270108    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"]
May 20 16:06:09.270594 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.270521    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq" podUID="f4ac65ba-473b-4af6-bfc2-bce8092a55ce" containerName="kserve-container" containerID="cri-o://625ef2131d5c4fbd5b4fc5854accea9d323510c3176893e399f9bfd51bc35e14" gracePeriod=30
May 20 16:06:09.270686 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.270660    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq" podUID="f4ac65ba-473b-4af6-bfc2-bce8092a55ce" containerName="kube-rbac-proxy" containerID="cri-o://25596ca59cb42cfa3940ef5718d17b398f535aee0e33689ffd3c1c63260f6d23" gracePeriod=30
May 20 16:06:09.300582 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.300550    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"]
May 20 16:06:09.301156 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.301139    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" containerName="kube-rbac-proxy"
May 20 16:06:09.301259 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.301159    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" containerName="kube-rbac-proxy"
May 20 16:06:09.301259 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.301175    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f6ac5b-7653-47eb-a82d-c64312fde5ee" containerName="kserve-container"
May 20 16:06:09.301259 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.301184    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f6ac5b-7653-47eb-a82d-c64312fde5ee" containerName="kserve-container"
May 20 16:06:09.301259 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.301194    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93c59413-c05b-43b0-b7d9-69de5d9d8344" containerName="ensemble-graph-3f928"
May 20 16:06:09.301259 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.301203    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="93c59413-c05b-43b0-b7d9-69de5d9d8344" containerName="ensemble-graph-3f928"
May 20 16:06:09.301259 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.301216    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" containerName="kserve-container"
May 20 16:06:09.301259 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.301224    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" containerName="kserve-container"
May 20 16:06:09.301259 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.301241    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f6ac5b-7653-47eb-a82d-c64312fde5ee" containerName="kube-rbac-proxy"
May 20 16:06:09.301259 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.301249    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f6ac5b-7653-47eb-a82d-c64312fde5ee" containerName="kube-rbac-proxy"
May 20 16:06:09.301724 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.301325    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" containerName="kserve-container"
May 20 16:06:09.301724 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.301336    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="93c59413-c05b-43b0-b7d9-69de5d9d8344" containerName="ensemble-graph-3f928"
May 20 16:06:09.301724 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.301348    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="62f6ac5b-7653-47eb-a82d-c64312fde5ee" containerName="kserve-container"
May 20 16:06:09.301724 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.301359    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="62f6ac5b-7653-47eb-a82d-c64312fde5ee" containerName="kube-rbac-proxy"
May 20 16:06:09.301724 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.301369    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="0a8d08a2-7d2b-4b3f-9064-8c59b18cab79" containerName="kube-rbac-proxy"
May 20 16:06:09.305267 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.305146    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"
May 20 16:06:09.307948 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.307926    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-62c44-predictor-serving-cert\""
May 20 16:06:09.307948 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.307942    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-62c44-kube-rbac-proxy-sar-config\""
May 20 16:06:09.314847 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.314819    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"]
May 20 16:06:09.342437 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.342406    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"]
May 20 16:06:09.342803 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.342773    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj" podUID="0eda94f7-0a45-42af-ace9-a99be2b1368c" containerName="kserve-container" containerID="cri-o://9b773dcf97ee087e6d326f9b6ddffb7e1c9a0999ec1e5fa3cd4d895504c2ad92" gracePeriod=30
May 20 16:06:09.342904 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.342838    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj" podUID="0eda94f7-0a45-42af-ace9-a99be2b1368c" containerName="kube-rbac-proxy" containerID="cri-o://f4c09c1c86924d46f0cba917f71e5dfb87249dff9bd90450fc54a6871f089e58" gracePeriod=30
May 20 16:06:09.406120 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.406078    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"]
May 20 16:06:09.409742 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.409719    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"
May 20 16:06:09.411147 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.411118    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh526\" (UniqueName: \"kubernetes.io/projected/ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed-kube-api-access-hh526\") pod \"success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp\" (UID: \"ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed\") " pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"
May 20 16:06:09.411276 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.411180    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed-proxy-tls\") pod \"success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp\" (UID: \"ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed\") " pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"
May 20 16:06:09.411330 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.411301    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-62c44-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed-success-200-isvc-62c44-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp\" (UID: \"ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed\") " pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"
May 20 16:06:09.412358 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.412336    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-62c44-predictor-serving-cert\""
May 20 16:06:09.412460 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.412355    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-62c44-kube-rbac-proxy-sar-config\""
May 20 16:06:09.417584 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.417538    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"]
May 20 16:06:09.512267 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.512234    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed-proxy-tls\") pod \"success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp\" (UID: \"ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed\") " pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"
May 20 16:06:09.512468 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.512303    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-62c44-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed-success-200-isvc-62c44-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp\" (UID: \"ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed\") " pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"
May 20 16:06:09.512468 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.512336    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92995157-1adc-4560-ae14-06cd7783fa6c-proxy-tls\") pod \"error-404-isvc-62c44-predictor-76f5b9974c-6fr6t\" (UID: \"92995157-1adc-4560-ae14-06cd7783fa6c\") " pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"
May 20 16:06:09.512468 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.512357    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-62c44-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92995157-1adc-4560-ae14-06cd7783fa6c-error-404-isvc-62c44-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-62c44-predictor-76f5b9974c-6fr6t\" (UID: \"92995157-1adc-4560-ae14-06cd7783fa6c\") " pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"
May 20 16:06:09.512468 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.512377    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q5pg\" (UniqueName: \"kubernetes.io/projected/92995157-1adc-4560-ae14-06cd7783fa6c-kube-api-access-8q5pg\") pod \"error-404-isvc-62c44-predictor-76f5b9974c-6fr6t\" (UID: \"92995157-1adc-4560-ae14-06cd7783fa6c\") " pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"
May 20 16:06:09.512468 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.512447    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hh526\" (UniqueName: \"kubernetes.io/projected/ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed-kube-api-access-hh526\") pod \"success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp\" (UID: \"ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed\") " pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"
May 20 16:06:09.512992 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.512971    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-62c44-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed-success-200-isvc-62c44-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp\" (UID: \"ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed\") " pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"
May 20 16:06:09.515154 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.515132    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed-proxy-tls\") pod \"success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp\" (UID: \"ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed\") " pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"
May 20 16:06:09.522049 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.521959    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh526\" (UniqueName: \"kubernetes.io/projected/ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed-kube-api-access-hh526\") pod \"success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp\" (UID: \"ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed\") " pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"
May 20 16:06:09.613044 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.612982    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92995157-1adc-4560-ae14-06cd7783fa6c-proxy-tls\") pod \"error-404-isvc-62c44-predictor-76f5b9974c-6fr6t\" (UID: \"92995157-1adc-4560-ae14-06cd7783fa6c\") " pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"
May 20 16:06:09.613242 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.613058    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-62c44-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92995157-1adc-4560-ae14-06cd7783fa6c-error-404-isvc-62c44-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-62c44-predictor-76f5b9974c-6fr6t\" (UID: \"92995157-1adc-4560-ae14-06cd7783fa6c\") " pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"
May 20 16:06:09.613242 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.613078    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8q5pg\" (UniqueName: \"kubernetes.io/projected/92995157-1adc-4560-ae14-06cd7783fa6c-kube-api-access-8q5pg\") pod \"error-404-isvc-62c44-predictor-76f5b9974c-6fr6t\" (UID: \"92995157-1adc-4560-ae14-06cd7783fa6c\") " pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"
May 20 16:06:09.613770 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.613737    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-62c44-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92995157-1adc-4560-ae14-06cd7783fa6c-error-404-isvc-62c44-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-62c44-predictor-76f5b9974c-6fr6t\" (UID: \"92995157-1adc-4560-ae14-06cd7783fa6c\") " pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"
May 20 16:06:09.615582 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.615559    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92995157-1adc-4560-ae14-06cd7783fa6c-proxy-tls\") pod \"error-404-isvc-62c44-predictor-76f5b9974c-6fr6t\" (UID: \"92995157-1adc-4560-ae14-06cd7783fa6c\") " pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"
May 20 16:06:09.620453 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.620424    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"
May 20 16:06:09.622894 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.622868    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q5pg\" (UniqueName: \"kubernetes.io/projected/92995157-1adc-4560-ae14-06cd7783fa6c-kube-api-access-8q5pg\") pod \"error-404-isvc-62c44-predictor-76f5b9974c-6fr6t\" (UID: \"92995157-1adc-4560-ae14-06cd7783fa6c\") " pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"
May 20 16:06:09.723229 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.723192    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"
May 20 16:06:09.760725 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.760695    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"]
May 20 16:06:09.763069 ip-10-0-129-117 kubenswrapper[2568]: W0520 16:06:09.763041    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc0ed77_16ae_45e9_b1e7_19f1c4b44fed.slice/crio-983738832d2557b41810e0f74611223399f6571d8a1b62d808d4721897ca0f24 WatchSource:0}: Error finding container 983738832d2557b41810e0f74611223399f6571d8a1b62d808d4721897ca0f24: Status 404 returned error can't find the container with id 983738832d2557b41810e0f74611223399f6571d8a1b62d808d4721897ca0f24
May 20 16:06:09.870369 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:09.870340    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"]
May 20 16:06:10.002404 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:10.002375    2568 generic.go:358] "Generic (PLEG): container finished" podID="f4ac65ba-473b-4af6-bfc2-bce8092a55ce" containerID="25596ca59cb42cfa3940ef5718d17b398f535aee0e33689ffd3c1c63260f6d23" exitCode=2
May 20 16:06:10.002559 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:10.002460    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq" event={"ID":"f4ac65ba-473b-4af6-bfc2-bce8092a55ce","Type":"ContainerDied","Data":"25596ca59cb42cfa3940ef5718d17b398f535aee0e33689ffd3c1c63260f6d23"}
May 20 16:06:10.004035 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:10.003982    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t" event={"ID":"92995157-1adc-4560-ae14-06cd7783fa6c","Type":"ContainerStarted","Data":"58bdb833e90cdc70f42cecbb123ee3e805952dc420fe93a7ab29a1042fd53d27"}
May 20 16:06:10.004137 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:10.004047    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t" event={"ID":"92995157-1adc-4560-ae14-06cd7783fa6c","Type":"ContainerStarted","Data":"f10955a70ede07ef7698555d825b8f641e86177d70d9eeadb2bcfb92e6356595"}
May 20 16:06:10.004137 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:10.004063    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t" event={"ID":"92995157-1adc-4560-ae14-06cd7783fa6c","Type":"ContainerStarted","Data":"27a05d3d2324dfc5d1c56621dfaf9a4832f088d0e66e016c48652e28d6adc9ae"}
May 20 16:06:10.004251 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:10.004192    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"
May 20 16:06:10.004251 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:10.004221    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"
May 20 16:06:10.005643 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:10.005618    2568 generic.go:358] "Generic (PLEG): container finished" podID="0eda94f7-0a45-42af-ace9-a99be2b1368c" containerID="f4c09c1c86924d46f0cba917f71e5dfb87249dff9bd90450fc54a6871f089e58" exitCode=2
May 20 16:06:10.005643 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:10.005631    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t" podUID="92995157-1adc-4560-ae14-06cd7783fa6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 20 16:06:10.005816 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:10.005686    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj" event={"ID":"0eda94f7-0a45-42af-ace9-a99be2b1368c","Type":"ContainerDied","Data":"f4c09c1c86924d46f0cba917f71e5dfb87249dff9bd90450fc54a6871f089e58"}
May 20 16:06:10.007077 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:10.007058    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp" event={"ID":"ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed","Type":"ContainerStarted","Data":"b25143c1c545d7dc63d47aaf10197ce9ea213a8f86e4ccce97bac93f1edf3be4"}
May 20 16:06:10.007186 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:10.007081    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp" event={"ID":"ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed","Type":"ContainerStarted","Data":"89ebe9197e8ec0ce91b984f553d5ee17c04fe97fadaec7f06a3b6039c23f004a"}
May 20 16:06:10.007186 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:10.007092    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp" event={"ID":"ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed","Type":"ContainerStarted","Data":"983738832d2557b41810e0f74611223399f6571d8a1b62d808d4721897ca0f24"}
May 20 16:06:10.007308 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:10.007295    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"
May 20 16:06:10.007361 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:10.007316    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"
May 20 16:06:10.008335 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:10.008311    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp" podUID="ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 20 16:06:10.025808 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:10.025713    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t" podStartSLOduration=1.025697469 podStartE2EDuration="1.025697469s" podCreationTimestamp="2026-05-20 16:06:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:06:10.023161925 +0000 UTC m=+2078.962003493" watchObservedRunningTime="2026-05-20 16:06:10.025697469 +0000 UTC m=+2078.964539025"
May 20 16:06:10.045743 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:10.045687    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp" podStartSLOduration=1.045672341 podStartE2EDuration="1.045672341s" podCreationTimestamp="2026-05-20 16:06:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:06:10.044278442 +0000 UTC m=+2078.983120001" watchObservedRunningTime="2026-05-20 16:06:10.045672341 +0000 UTC m=+2078.984513897"
May 20 16:06:11.012079 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:11.012009    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t" podUID="92995157-1adc-4560-ae14-06cd7783fa6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 20 16:06:11.012497 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:11.012135    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp" podUID="ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 20 16:06:11.694768 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:11.694723    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq" podUID="f4ac65ba-473b-4af6-bfc2-bce8092a55ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
May 20 16:06:12.701831 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:12.701787    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj" podUID="0eda94f7-0a45-42af-ace9-a99be2b1368c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
May 20 16:06:12.867839 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:12.867815    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"
May 20 16:06:12.993552 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:12.993530    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"
May 20 16:06:13.021409 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.021373    2568 generic.go:358] "Generic (PLEG): container finished" podID="0eda94f7-0a45-42af-ace9-a99be2b1368c" containerID="9b773dcf97ee087e6d326f9b6ddffb7e1c9a0999ec1e5fa3cd4d895504c2ad92" exitCode=0
May 20 16:06:13.021581 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.021447    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj" event={"ID":"0eda94f7-0a45-42af-ace9-a99be2b1368c","Type":"ContainerDied","Data":"9b773dcf97ee087e6d326f9b6ddffb7e1c9a0999ec1e5fa3cd4d895504c2ad92"}
May 20 16:06:13.021581 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.021485    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj" event={"ID":"0eda94f7-0a45-42af-ace9-a99be2b1368c","Type":"ContainerDied","Data":"48137aba873bf35a4b6d8f7dbba7dc10957a452111686f0e4ed9a974528ea980"}
May 20 16:06:13.021581 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.021456    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"
May 20 16:06:13.021581 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.021501    2568 scope.go:117] "RemoveContainer" containerID="f4c09c1c86924d46f0cba917f71e5dfb87249dff9bd90450fc54a6871f089e58"
May 20 16:06:13.023213 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.023189    2568 generic.go:358] "Generic (PLEG): container finished" podID="f4ac65ba-473b-4af6-bfc2-bce8092a55ce" containerID="625ef2131d5c4fbd5b4fc5854accea9d323510c3176893e399f9bfd51bc35e14" exitCode=0
May 20 16:06:13.023342 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.023247    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq" event={"ID":"f4ac65ba-473b-4af6-bfc2-bce8092a55ce","Type":"ContainerDied","Data":"625ef2131d5c4fbd5b4fc5854accea9d323510c3176893e399f9bfd51bc35e14"}
May 20 16:06:13.023342 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.023272    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq" event={"ID":"f4ac65ba-473b-4af6-bfc2-bce8092a55ce","Type":"ContainerDied","Data":"28fd16f5335dc13bad45fa5891c58bcc15af13e9f208a796c40d43ddcadba63b"}
May 20 16:06:13.023342 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.023279    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"
May 20 16:06:13.030239 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.030219    2568 scope.go:117] "RemoveContainer" containerID="9b773dcf97ee087e6d326f9b6ddffb7e1c9a0999ec1e5fa3cd4d895504c2ad92"
May 20 16:06:13.040548 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.040528    2568 scope.go:117] "RemoveContainer" containerID="f4c09c1c86924d46f0cba917f71e5dfb87249dff9bd90450fc54a6871f089e58"
May 20 16:06:13.040867 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:06:13.040837    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4c09c1c86924d46f0cba917f71e5dfb87249dff9bd90450fc54a6871f089e58\": container with ID starting with f4c09c1c86924d46f0cba917f71e5dfb87249dff9bd90450fc54a6871f089e58 not found: ID does not exist" containerID="f4c09c1c86924d46f0cba917f71e5dfb87249dff9bd90450fc54a6871f089e58"
May 20 16:06:13.040932 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.040883    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c09c1c86924d46f0cba917f71e5dfb87249dff9bd90450fc54a6871f089e58"} err="failed to get container status \"f4c09c1c86924d46f0cba917f71e5dfb87249dff9bd90450fc54a6871f089e58\": rpc error: code = NotFound desc = could not find container \"f4c09c1c86924d46f0cba917f71e5dfb87249dff9bd90450fc54a6871f089e58\": container with ID starting with f4c09c1c86924d46f0cba917f71e5dfb87249dff9bd90450fc54a6871f089e58 not found: ID does not exist"
May 20 16:06:13.040932 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.040910    2568 scope.go:117] "RemoveContainer" containerID="9b773dcf97ee087e6d326f9b6ddffb7e1c9a0999ec1e5fa3cd4d895504c2ad92"
May 20 16:06:13.041187 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:06:13.041168    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b773dcf97ee087e6d326f9b6ddffb7e1c9a0999ec1e5fa3cd4d895504c2ad92\": container with ID starting with 9b773dcf97ee087e6d326f9b6ddffb7e1c9a0999ec1e5fa3cd4d895504c2ad92 not found: ID does not exist" containerID="9b773dcf97ee087e6d326f9b6ddffb7e1c9a0999ec1e5fa3cd4d895504c2ad92"
May 20 16:06:13.041250 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.041193    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b773dcf97ee087e6d326f9b6ddffb7e1c9a0999ec1e5fa3cd4d895504c2ad92"} err="failed to get container status \"9b773dcf97ee087e6d326f9b6ddffb7e1c9a0999ec1e5fa3cd4d895504c2ad92\": rpc error: code = NotFound desc = could not find container \"9b773dcf97ee087e6d326f9b6ddffb7e1c9a0999ec1e5fa3cd4d895504c2ad92\": container with ID starting with 9b773dcf97ee087e6d326f9b6ddffb7e1c9a0999ec1e5fa3cd4d895504c2ad92 not found: ID does not exist"
May 20 16:06:13.041250 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.041219    2568 scope.go:117] "RemoveContainer" containerID="25596ca59cb42cfa3940ef5718d17b398f535aee0e33689ffd3c1c63260f6d23"
May 20 16:06:13.043255 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.043230    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-e05eb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4ac65ba-473b-4af6-bfc2-bce8092a55ce-success-200-isvc-e05eb-kube-rbac-proxy-sar-config\") pod \"f4ac65ba-473b-4af6-bfc2-bce8092a55ce\" (UID: \"f4ac65ba-473b-4af6-bfc2-bce8092a55ce\") "
May 20 16:06:13.043346 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.043326    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrq88\" (UniqueName: \"kubernetes.io/projected/f4ac65ba-473b-4af6-bfc2-bce8092a55ce-kube-api-access-wrq88\") pod \"f4ac65ba-473b-4af6-bfc2-bce8092a55ce\" (UID: \"f4ac65ba-473b-4af6-bfc2-bce8092a55ce\") "
May 20 16:06:13.043388 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.043368    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4ac65ba-473b-4af6-bfc2-bce8092a55ce-proxy-tls\") pod \"f4ac65ba-473b-4af6-bfc2-bce8092a55ce\" (UID: \"f4ac65ba-473b-4af6-bfc2-bce8092a55ce\") "
May 20 16:06:13.043523 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.043502    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ac65ba-473b-4af6-bfc2-bce8092a55ce-success-200-isvc-e05eb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-e05eb-kube-rbac-proxy-sar-config") pod "f4ac65ba-473b-4af6-bfc2-bce8092a55ce" (UID: "f4ac65ba-473b-4af6-bfc2-bce8092a55ce"). InnerVolumeSpecName "success-200-isvc-e05eb-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:06:13.043719 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.043695    2568 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-e05eb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4ac65ba-473b-4af6-bfc2-bce8092a55ce-success-200-isvc-e05eb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:06:13.045473 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.045445    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4ac65ba-473b-4af6-bfc2-bce8092a55ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f4ac65ba-473b-4af6-bfc2-bce8092a55ce" (UID: "f4ac65ba-473b-4af6-bfc2-bce8092a55ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:06:13.045545 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.045515    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ac65ba-473b-4af6-bfc2-bce8092a55ce-kube-api-access-wrq88" (OuterVolumeSpecName: "kube-api-access-wrq88") pod "f4ac65ba-473b-4af6-bfc2-bce8092a55ce" (UID: "f4ac65ba-473b-4af6-bfc2-bce8092a55ce"). InnerVolumeSpecName "kube-api-access-wrq88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:06:13.049264 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.049243    2568 scope.go:117] "RemoveContainer" containerID="625ef2131d5c4fbd5b4fc5854accea9d323510c3176893e399f9bfd51bc35e14"
May 20 16:06:13.056454 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.056432    2568 scope.go:117] "RemoveContainer" containerID="25596ca59cb42cfa3940ef5718d17b398f535aee0e33689ffd3c1c63260f6d23"
May 20 16:06:13.056727 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:06:13.056706    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25596ca59cb42cfa3940ef5718d17b398f535aee0e33689ffd3c1c63260f6d23\": container with ID starting with 25596ca59cb42cfa3940ef5718d17b398f535aee0e33689ffd3c1c63260f6d23 not found: ID does not exist" containerID="25596ca59cb42cfa3940ef5718d17b398f535aee0e33689ffd3c1c63260f6d23"
May 20 16:06:13.056809 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.056738    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25596ca59cb42cfa3940ef5718d17b398f535aee0e33689ffd3c1c63260f6d23"} err="failed to get container status \"25596ca59cb42cfa3940ef5718d17b398f535aee0e33689ffd3c1c63260f6d23\": rpc error: code = NotFound desc = could not find container \"25596ca59cb42cfa3940ef5718d17b398f535aee0e33689ffd3c1c63260f6d23\": container with ID starting with 25596ca59cb42cfa3940ef5718d17b398f535aee0e33689ffd3c1c63260f6d23 not found: ID does not exist"
May 20 16:06:13.056809 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.056763    2568 scope.go:117] "RemoveContainer" containerID="625ef2131d5c4fbd5b4fc5854accea9d323510c3176893e399f9bfd51bc35e14"
May 20 16:06:13.057150 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:06:13.057004    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"625ef2131d5c4fbd5b4fc5854accea9d323510c3176893e399f9bfd51bc35e14\": container with ID starting with 625ef2131d5c4fbd5b4fc5854accea9d323510c3176893e399f9bfd51bc35e14 not found: ID does not exist" containerID="625ef2131d5c4fbd5b4fc5854accea9d323510c3176893e399f9bfd51bc35e14"
May 20 16:06:13.057150 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.057043    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"625ef2131d5c4fbd5b4fc5854accea9d323510c3176893e399f9bfd51bc35e14"} err="failed to get container status \"625ef2131d5c4fbd5b4fc5854accea9d323510c3176893e399f9bfd51bc35e14\": rpc error: code = NotFound desc = could not find container \"625ef2131d5c4fbd5b4fc5854accea9d323510c3176893e399f9bfd51bc35e14\": container with ID starting with 625ef2131d5c4fbd5b4fc5854accea9d323510c3176893e399f9bfd51bc35e14 not found: ID does not exist"
May 20 16:06:13.144423 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.144327    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bphhl\" (UniqueName: \"kubernetes.io/projected/0eda94f7-0a45-42af-ace9-a99be2b1368c-kube-api-access-bphhl\") pod \"0eda94f7-0a45-42af-ace9-a99be2b1368c\" (UID: \"0eda94f7-0a45-42af-ace9-a99be2b1368c\") "
May 20 16:06:13.144423 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.144374    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-e05eb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0eda94f7-0a45-42af-ace9-a99be2b1368c-error-404-isvc-e05eb-kube-rbac-proxy-sar-config\") pod \"0eda94f7-0a45-42af-ace9-a99be2b1368c\" (UID: \"0eda94f7-0a45-42af-ace9-a99be2b1368c\") "
May 20 16:06:13.144423 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.144408    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eda94f7-0a45-42af-ace9-a99be2b1368c-proxy-tls\") pod \"0eda94f7-0a45-42af-ace9-a99be2b1368c\" (UID: \"0eda94f7-0a45-42af-ace9-a99be2b1368c\") "
May 20 16:06:13.144670 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.144651    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrq88\" (UniqueName: \"kubernetes.io/projected/f4ac65ba-473b-4af6-bfc2-bce8092a55ce-kube-api-access-wrq88\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:06:13.144710 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.144672    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4ac65ba-473b-4af6-bfc2-bce8092a55ce-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:06:13.144749 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.144718    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eda94f7-0a45-42af-ace9-a99be2b1368c-error-404-isvc-e05eb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-e05eb-kube-rbac-proxy-sar-config") pod "0eda94f7-0a45-42af-ace9-a99be2b1368c" (UID: "0eda94f7-0a45-42af-ace9-a99be2b1368c"). InnerVolumeSpecName "error-404-isvc-e05eb-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:06:13.146603 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.146580    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eda94f7-0a45-42af-ace9-a99be2b1368c-kube-api-access-bphhl" (OuterVolumeSpecName: "kube-api-access-bphhl") pod "0eda94f7-0a45-42af-ace9-a99be2b1368c" (UID: "0eda94f7-0a45-42af-ace9-a99be2b1368c"). InnerVolumeSpecName "kube-api-access-bphhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:06:13.146674 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.146631    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eda94f7-0a45-42af-ace9-a99be2b1368c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0eda94f7-0a45-42af-ace9-a99be2b1368c" (UID: "0eda94f7-0a45-42af-ace9-a99be2b1368c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:06:13.245460 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.245410    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bphhl\" (UniqueName: \"kubernetes.io/projected/0eda94f7-0a45-42af-ace9-a99be2b1368c-kube-api-access-bphhl\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:06:13.245460 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.245453    2568 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-e05eb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0eda94f7-0a45-42af-ace9-a99be2b1368c-error-404-isvc-e05eb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:06:13.245460 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.245466    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eda94f7-0a45-42af-ace9-a99be2b1368c-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:06:13.349572 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.349520    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"]
May 20 16:06:13.354624 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.354600    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e05eb-predictor-7cf65457c5-6cgfj"]
May 20 16:06:13.367085 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.367052    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"]
May 20 16:06:13.376221 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.376191    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e05eb-predictor-d89f8c48d-wrdnq"]
May 20 16:06:13.644901 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.644869    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eda94f7-0a45-42af-ace9-a99be2b1368c" path="/var/lib/kubelet/pods/0eda94f7-0a45-42af-ace9-a99be2b1368c/volumes"
May 20 16:06:13.645360 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:13.645345    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4ac65ba-473b-4af6-bfc2-bce8092a55ce" path="/var/lib/kubelet/pods/f4ac65ba-473b-4af6-bfc2-bce8092a55ce/volumes"
May 20 16:06:16.017234 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:16.017204    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"
May 20 16:06:16.017698 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:16.017262    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"
May 20 16:06:16.017698 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:16.017631    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t" podUID="92995157-1adc-4560-ae14-06cd7783fa6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 20 16:06:16.017825 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:16.017638    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp" podUID="ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 20 16:06:26.018627 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:26.018586    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp" podUID="ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 20 16:06:26.019154 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:26.018586    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t" podUID="92995157-1adc-4560-ae14-06cd7783fa6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 20 16:06:31.727230 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:31.727198    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 16:06:31.730428 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:31.730401    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 16:06:36.017719 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:36.017677    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t" podUID="92995157-1adc-4560-ae14-06cd7783fa6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 20 16:06:36.018228 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:36.017677    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp" podUID="ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 20 16:06:46.017970 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:46.017928    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp" podUID="ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 20 16:06:46.018385 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:46.017946    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t" podUID="92995157-1adc-4560-ae14-06cd7783fa6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 20 16:06:56.019009 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:56.018976    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"
May 20 16:06:56.019533 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:06:56.019057    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"
May 20 16:11:31.752223 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:11:31.752196    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 16:11:31.756721 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:11:31.756695    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 16:15:24.208366 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:24.208317    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"]
May 20 16:15:24.208845 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:24.208652    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp" podUID="ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed" containerName="kserve-container" containerID="cri-o://89ebe9197e8ec0ce91b984f553d5ee17c04fe97fadaec7f06a3b6039c23f004a" gracePeriod=30
May 20 16:15:24.208845 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:24.208672    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp" podUID="ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed" containerName="kube-rbac-proxy" containerID="cri-o://b25143c1c545d7dc63d47aaf10197ce9ea213a8f86e4ccce97bac93f1edf3be4" gracePeriod=30
May 20 16:15:24.289226 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:24.289179    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"]
May 20 16:15:24.289495 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:24.289474    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t" podUID="92995157-1adc-4560-ae14-06cd7783fa6c" containerName="kserve-container" containerID="cri-o://f10955a70ede07ef7698555d825b8f641e86177d70d9eeadb2bcfb92e6356595" gracePeriod=30
May 20 16:15:24.289586 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:24.289553    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t" podUID="92995157-1adc-4560-ae14-06cd7783fa6c" containerName="kube-rbac-proxy" containerID="cri-o://58bdb833e90cdc70f42cecbb123ee3e805952dc420fe93a7ab29a1042fd53d27" gracePeriod=30
May 20 16:15:24.958407 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:24.958375    2568 generic.go:358] "Generic (PLEG): container finished" podID="ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed" containerID="b25143c1c545d7dc63d47aaf10197ce9ea213a8f86e4ccce97bac93f1edf3be4" exitCode=2
May 20 16:15:24.958610 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:24.958450    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp" event={"ID":"ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed","Type":"ContainerDied","Data":"b25143c1c545d7dc63d47aaf10197ce9ea213a8f86e4ccce97bac93f1edf3be4"}
May 20 16:15:24.959957 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:24.959936    2568 generic.go:358] "Generic (PLEG): container finished" podID="92995157-1adc-4560-ae14-06cd7783fa6c" containerID="58bdb833e90cdc70f42cecbb123ee3e805952dc420fe93a7ab29a1042fd53d27" exitCode=2
May 20 16:15:24.960097 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:24.960005    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t" event={"ID":"92995157-1adc-4560-ae14-06cd7783fa6c","Type":"ContainerDied","Data":"58bdb833e90cdc70f42cecbb123ee3e805952dc420fe93a7ab29a1042fd53d27"}
May 20 16:15:26.012663 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:26.012619    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t" podUID="92995157-1adc-4560-ae14-06cd7783fa6c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": dial tcp 10.132.0.62:8643: connect: connection refused"
May 20 16:15:26.013076 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:26.012629    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp" podUID="ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.61:8643/healthz\": dial tcp 10.132.0.61:8643: connect: connection refused"
May 20 16:15:26.017980 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:26.017955    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp" podUID="ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 20 16:15:26.017980 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:26.017974    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t" podUID="92995157-1adc-4560-ae14-06cd7783fa6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 20 16:15:27.663996 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.663960    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"
May 20 16:15:27.759733 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.759706    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed-proxy-tls\") pod \"ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed\" (UID: \"ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed\") "
May 20 16:15:27.759868 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.759744    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-62c44-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed-success-200-isvc-62c44-kube-rbac-proxy-sar-config\") pod \"ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed\" (UID: \"ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed\") "
May 20 16:15:27.759868 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.759787    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh526\" (UniqueName: \"kubernetes.io/projected/ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed-kube-api-access-hh526\") pod \"ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed\" (UID: \"ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed\") "
May 20 16:15:27.760206 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.760174    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed-success-200-isvc-62c44-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-62c44-kube-rbac-proxy-sar-config") pod "ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed" (UID: "ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed"). InnerVolumeSpecName "success-200-isvc-62c44-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:15:27.762142 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.762122    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"
May 20 16:15:27.762268 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.762247    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed" (UID: "ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:15:27.762324 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.762273    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed-kube-api-access-hh526" (OuterVolumeSpecName: "kube-api-access-hh526") pod "ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed" (UID: "ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed"). InnerVolumeSpecName "kube-api-access-hh526". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:15:27.860855 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.860820    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8q5pg\" (UniqueName: \"kubernetes.io/projected/92995157-1adc-4560-ae14-06cd7783fa6c-kube-api-access-8q5pg\") pod \"92995157-1adc-4560-ae14-06cd7783fa6c\" (UID: \"92995157-1adc-4560-ae14-06cd7783fa6c\") "
May 20 16:15:27.861101 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.860884    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92995157-1adc-4560-ae14-06cd7783fa6c-proxy-tls\") pod \"92995157-1adc-4560-ae14-06cd7783fa6c\" (UID: \"92995157-1adc-4560-ae14-06cd7783fa6c\") "
May 20 16:15:27.861101 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.860947    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-62c44-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92995157-1adc-4560-ae14-06cd7783fa6c-error-404-isvc-62c44-kube-rbac-proxy-sar-config\") pod \"92995157-1adc-4560-ae14-06cd7783fa6c\" (UID: \"92995157-1adc-4560-ae14-06cd7783fa6c\") "
May 20 16:15:27.861243 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.861167    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:15:27.861243 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.861179    2568 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-62c44-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed-success-200-isvc-62c44-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:15:27.861243 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.861189    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh526\" (UniqueName: \"kubernetes.io/projected/ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed-kube-api-access-hh526\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:15:27.861373 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.861317    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92995157-1adc-4560-ae14-06cd7783fa6c-error-404-isvc-62c44-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-62c44-kube-rbac-proxy-sar-config") pod "92995157-1adc-4560-ae14-06cd7783fa6c" (UID: "92995157-1adc-4560-ae14-06cd7783fa6c"). InnerVolumeSpecName "error-404-isvc-62c44-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:15:27.863186 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.863162    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92995157-1adc-4560-ae14-06cd7783fa6c-kube-api-access-8q5pg" (OuterVolumeSpecName: "kube-api-access-8q5pg") pod "92995157-1adc-4560-ae14-06cd7783fa6c" (UID: "92995157-1adc-4560-ae14-06cd7783fa6c"). InnerVolumeSpecName "kube-api-access-8q5pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:15:27.863269 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.863167    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92995157-1adc-4560-ae14-06cd7783fa6c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "92995157-1adc-4560-ae14-06cd7783fa6c" (UID: "92995157-1adc-4560-ae14-06cd7783fa6c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:15:27.961825 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.961783    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q5pg\" (UniqueName: \"kubernetes.io/projected/92995157-1adc-4560-ae14-06cd7783fa6c-kube-api-access-8q5pg\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:15:27.961825 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.961817    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92995157-1adc-4560-ae14-06cd7783fa6c-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:15:27.961825 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.961827    2568 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-62c44-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92995157-1adc-4560-ae14-06cd7783fa6c-error-404-isvc-62c44-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:15:27.971882 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.971851    2568 generic.go:358] "Generic (PLEG): container finished" podID="92995157-1adc-4560-ae14-06cd7783fa6c" containerID="f10955a70ede07ef7698555d825b8f641e86177d70d9eeadb2bcfb92e6356595" exitCode=0
May 20 16:15:27.972035 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.971924    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"
May 20 16:15:27.972035 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.971938    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t" event={"ID":"92995157-1adc-4560-ae14-06cd7783fa6c","Type":"ContainerDied","Data":"f10955a70ede07ef7698555d825b8f641e86177d70d9eeadb2bcfb92e6356595"}
May 20 16:15:27.972035 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.971978    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t" event={"ID":"92995157-1adc-4560-ae14-06cd7783fa6c","Type":"ContainerDied","Data":"27a05d3d2324dfc5d1c56621dfaf9a4832f088d0e66e016c48652e28d6adc9ae"}
May 20 16:15:27.972035 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.971996    2568 scope.go:117] "RemoveContainer" containerID="58bdb833e90cdc70f42cecbb123ee3e805952dc420fe93a7ab29a1042fd53d27"
May 20 16:15:27.973567 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.973542    2568 generic.go:358] "Generic (PLEG): container finished" podID="ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed" containerID="89ebe9197e8ec0ce91b984f553d5ee17c04fe97fadaec7f06a3b6039c23f004a" exitCode=0
May 20 16:15:27.973685 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.973601    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp" event={"ID":"ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed","Type":"ContainerDied","Data":"89ebe9197e8ec0ce91b984f553d5ee17c04fe97fadaec7f06a3b6039c23f004a"}
May 20 16:15:27.973685 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.973615    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"
May 20 16:15:27.973685 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.973625    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp" event={"ID":"ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed","Type":"ContainerDied","Data":"983738832d2557b41810e0f74611223399f6571d8a1b62d808d4721897ca0f24"}
May 20 16:15:27.982381 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.982355    2568 scope.go:117] "RemoveContainer" containerID="f10955a70ede07ef7698555d825b8f641e86177d70d9eeadb2bcfb92e6356595"
May 20 16:15:27.990257 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.990237    2568 scope.go:117] "RemoveContainer" containerID="58bdb833e90cdc70f42cecbb123ee3e805952dc420fe93a7ab29a1042fd53d27"
May 20 16:15:27.990535 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:15:27.990514    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58bdb833e90cdc70f42cecbb123ee3e805952dc420fe93a7ab29a1042fd53d27\": container with ID starting with 58bdb833e90cdc70f42cecbb123ee3e805952dc420fe93a7ab29a1042fd53d27 not found: ID does not exist" containerID="58bdb833e90cdc70f42cecbb123ee3e805952dc420fe93a7ab29a1042fd53d27"
May 20 16:15:27.990610 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.990548    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58bdb833e90cdc70f42cecbb123ee3e805952dc420fe93a7ab29a1042fd53d27"} err="failed to get container status \"58bdb833e90cdc70f42cecbb123ee3e805952dc420fe93a7ab29a1042fd53d27\": rpc error: code = NotFound desc = could not find container \"58bdb833e90cdc70f42cecbb123ee3e805952dc420fe93a7ab29a1042fd53d27\": container with ID starting with 58bdb833e90cdc70f42cecbb123ee3e805952dc420fe93a7ab29a1042fd53d27 not found: ID does not exist"
May 20 16:15:27.990610 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.990573    2568 scope.go:117] "RemoveContainer" containerID="f10955a70ede07ef7698555d825b8f641e86177d70d9eeadb2bcfb92e6356595"
May 20 16:15:27.990818 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:15:27.990799    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f10955a70ede07ef7698555d825b8f641e86177d70d9eeadb2bcfb92e6356595\": container with ID starting with f10955a70ede07ef7698555d825b8f641e86177d70d9eeadb2bcfb92e6356595 not found: ID does not exist" containerID="f10955a70ede07ef7698555d825b8f641e86177d70d9eeadb2bcfb92e6356595"
May 20 16:15:27.990854 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.990825    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f10955a70ede07ef7698555d825b8f641e86177d70d9eeadb2bcfb92e6356595"} err="failed to get container status \"f10955a70ede07ef7698555d825b8f641e86177d70d9eeadb2bcfb92e6356595\": rpc error: code = NotFound desc = could not find container \"f10955a70ede07ef7698555d825b8f641e86177d70d9eeadb2bcfb92e6356595\": container with ID starting with f10955a70ede07ef7698555d825b8f641e86177d70d9eeadb2bcfb92e6356595 not found: ID does not exist"
May 20 16:15:27.990854 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.990841    2568 scope.go:117] "RemoveContainer" containerID="b25143c1c545d7dc63d47aaf10197ce9ea213a8f86e4ccce97bac93f1edf3be4"
May 20 16:15:27.999105 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.999080    2568 scope.go:117] "RemoveContainer" containerID="89ebe9197e8ec0ce91b984f553d5ee17c04fe97fadaec7f06a3b6039c23f004a"
May 20 16:15:27.999806 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:27.999753    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"]
May 20 16:15:28.004311 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:28.004288    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-62c44-predictor-76f5b9974c-6fr6t"]
May 20 16:15:28.009217 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:28.009199    2568 scope.go:117] "RemoveContainer" containerID="b25143c1c545d7dc63d47aaf10197ce9ea213a8f86e4ccce97bac93f1edf3be4"
May 20 16:15:28.009476 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:15:28.009459    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b25143c1c545d7dc63d47aaf10197ce9ea213a8f86e4ccce97bac93f1edf3be4\": container with ID starting with b25143c1c545d7dc63d47aaf10197ce9ea213a8f86e4ccce97bac93f1edf3be4 not found: ID does not exist" containerID="b25143c1c545d7dc63d47aaf10197ce9ea213a8f86e4ccce97bac93f1edf3be4"
May 20 16:15:28.009525 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:28.009485    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b25143c1c545d7dc63d47aaf10197ce9ea213a8f86e4ccce97bac93f1edf3be4"} err="failed to get container status \"b25143c1c545d7dc63d47aaf10197ce9ea213a8f86e4ccce97bac93f1edf3be4\": rpc error: code = NotFound desc = could not find container \"b25143c1c545d7dc63d47aaf10197ce9ea213a8f86e4ccce97bac93f1edf3be4\": container with ID starting with b25143c1c545d7dc63d47aaf10197ce9ea213a8f86e4ccce97bac93f1edf3be4 not found: ID does not exist"
May 20 16:15:28.009525 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:28.009505    2568 scope.go:117] "RemoveContainer" containerID="89ebe9197e8ec0ce91b984f553d5ee17c04fe97fadaec7f06a3b6039c23f004a"
May 20 16:15:28.009798 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:15:28.009779    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ebe9197e8ec0ce91b984f553d5ee17c04fe97fadaec7f06a3b6039c23f004a\": container with ID starting with 89ebe9197e8ec0ce91b984f553d5ee17c04fe97fadaec7f06a3b6039c23f004a not found: ID does not exist" containerID="89ebe9197e8ec0ce91b984f553d5ee17c04fe97fadaec7f06a3b6039c23f004a"
May 20 16:15:28.009852 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:28.009805    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ebe9197e8ec0ce91b984f553d5ee17c04fe97fadaec7f06a3b6039c23f004a"} err="failed to get container status \"89ebe9197e8ec0ce91b984f553d5ee17c04fe97fadaec7f06a3b6039c23f004a\": rpc error: code = NotFound desc = could not find container \"89ebe9197e8ec0ce91b984f553d5ee17c04fe97fadaec7f06a3b6039c23f004a\": container with ID starting with 89ebe9197e8ec0ce91b984f553d5ee17c04fe97fadaec7f06a3b6039c23f004a not found: ID does not exist"
May 20 16:15:28.014655 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:28.014634    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"]
May 20 16:15:28.020321 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:28.020298    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-62c44-predictor-7c56bb66c6-6jdmp"]
May 20 16:15:29.643633 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:29.643600    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92995157-1adc-4560-ae14-06cd7783fa6c" path="/var/lib/kubelet/pods/92995157-1adc-4560-ae14-06cd7783fa6c/volumes"
May 20 16:15:29.644008 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:15:29.643998    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed" path="/var/lib/kubelet/pods/ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed/volumes"
May 20 16:16:31.776944 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:16:31.776912    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 16:16:31.781359 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:16:31.781336    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 16:21:31.800772 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:21:31.800743    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 16:21:31.806655 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:21:31.806635    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 16:22:34.233902 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:34.233868    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"]
May 20 16:22:34.234447 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:34.234172    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56" podUID="722a5aa0-74ba-479e-91d4-e5db4b6b91e4" containerName="kserve-container" containerID="cri-o://4d3d68a20fc9274367ea66fb3f9a8a2148f849f030fd63743e569e10f1197ed2" gracePeriod=30
May 20 16:22:34.234447 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:34.234205    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56" podUID="722a5aa0-74ba-479e-91d4-e5db4b6b91e4" containerName="kube-rbac-proxy" containerID="cri-o://ee01c263670252341de6fa275890ae9fd5b017badf97090fe6879b0c77d7fa06" gracePeriod=30
May 20 16:22:34.291647 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:34.291620    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"]
May 20 16:22:34.291916 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:34.291891    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24" podUID="84084650-0c60-412c-af89-602e1b8a8101" containerName="kserve-container" containerID="cri-o://859cd27f334a62a8392894775dd886b43e3b9ee7efb9513358d9adc9fd683890" gracePeriod=30
May 20 16:22:34.292003 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:34.291949    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24" podUID="84084650-0c60-412c-af89-602e1b8a8101" containerName="kube-rbac-proxy" containerID="cri-o://9679969d6747a4ab2bce30dcd8ed1a01a3bc7167f684a9ece4539c5e462a72f8" gracePeriod=30
May 20 16:22:34.434411 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:34.434381    2568 generic.go:358] "Generic (PLEG): container finished" podID="722a5aa0-74ba-479e-91d4-e5db4b6b91e4" containerID="ee01c263670252341de6fa275890ae9fd5b017badf97090fe6879b0c77d7fa06" exitCode=2
May 20 16:22:34.434555 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:34.434460    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56" event={"ID":"722a5aa0-74ba-479e-91d4-e5db4b6b91e4","Type":"ContainerDied","Data":"ee01c263670252341de6fa275890ae9fd5b017badf97090fe6879b0c77d7fa06"}
May 20 16:22:34.435903 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:34.435884    2568 generic.go:358] "Generic (PLEG): container finished" podID="84084650-0c60-412c-af89-602e1b8a8101" containerID="9679969d6747a4ab2bce30dcd8ed1a01a3bc7167f684a9ece4539c5e462a72f8" exitCode=2
May 20 16:22:34.436014 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:34.435908    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24" event={"ID":"84084650-0c60-412c-af89-602e1b8a8101","Type":"ContainerDied","Data":"9679969d6747a4ab2bce30dcd8ed1a01a3bc7167f684a9ece4539c5e462a72f8"}
May 20 16:22:37.236260 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.236194    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"
May 20 16:22:37.315650 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.315624    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtc58\" (UniqueName: \"kubernetes.io/projected/722a5aa0-74ba-479e-91d4-e5db4b6b91e4-kube-api-access-qtc58\") pod \"722a5aa0-74ba-479e-91d4-e5db4b6b91e4\" (UID: \"722a5aa0-74ba-479e-91d4-e5db4b6b91e4\") "
May 20 16:22:37.315823 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.315677    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/722a5aa0-74ba-479e-91d4-e5db4b6b91e4-proxy-tls\") pod \"722a5aa0-74ba-479e-91d4-e5db4b6b91e4\" (UID: \"722a5aa0-74ba-479e-91d4-e5db4b6b91e4\") "
May 20 16:22:37.315823 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.315736    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-fd7d8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/722a5aa0-74ba-479e-91d4-e5db4b6b91e4-success-200-isvc-fd7d8-kube-rbac-proxy-sar-config\") pod \"722a5aa0-74ba-479e-91d4-e5db4b6b91e4\" (UID: \"722a5aa0-74ba-479e-91d4-e5db4b6b91e4\") "
May 20 16:22:37.316184 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.316160    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/722a5aa0-74ba-479e-91d4-e5db4b6b91e4-success-200-isvc-fd7d8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-fd7d8-kube-rbac-proxy-sar-config") pod "722a5aa0-74ba-479e-91d4-e5db4b6b91e4" (UID: "722a5aa0-74ba-479e-91d4-e5db4b6b91e4"). InnerVolumeSpecName "success-200-isvc-fd7d8-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:22:37.317879 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.317857    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722a5aa0-74ba-479e-91d4-e5db4b6b91e4-kube-api-access-qtc58" (OuterVolumeSpecName: "kube-api-access-qtc58") pod "722a5aa0-74ba-479e-91d4-e5db4b6b91e4" (UID: "722a5aa0-74ba-479e-91d4-e5db4b6b91e4"). InnerVolumeSpecName "kube-api-access-qtc58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:22:37.317954 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.317906    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/722a5aa0-74ba-479e-91d4-e5db4b6b91e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "722a5aa0-74ba-479e-91d4-e5db4b6b91e4" (UID: "722a5aa0-74ba-479e-91d4-e5db4b6b91e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:22:37.333882 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.333865    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"
May 20 16:22:37.416350 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.416324    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvwxk\" (UniqueName: \"kubernetes.io/projected/84084650-0c60-412c-af89-602e1b8a8101-kube-api-access-zvwxk\") pod \"84084650-0c60-412c-af89-602e1b8a8101\" (UID: \"84084650-0c60-412c-af89-602e1b8a8101\") "
May 20 16:22:37.416482 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.416361    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84084650-0c60-412c-af89-602e1b8a8101-proxy-tls\") pod \"84084650-0c60-412c-af89-602e1b8a8101\" (UID: \"84084650-0c60-412c-af89-602e1b8a8101\") "
May 20 16:22:37.416482 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.416424    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-fd7d8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84084650-0c60-412c-af89-602e1b8a8101-error-404-isvc-fd7d8-kube-rbac-proxy-sar-config\") pod \"84084650-0c60-412c-af89-602e1b8a8101\" (UID: \"84084650-0c60-412c-af89-602e1b8a8101\") "
May 20 16:22:37.416563 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.416541    2568 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-fd7d8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/722a5aa0-74ba-479e-91d4-e5db4b6b91e4-success-200-isvc-fd7d8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:22:37.416563 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.416552    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtc58\" (UniqueName: \"kubernetes.io/projected/722a5aa0-74ba-479e-91d4-e5db4b6b91e4-kube-api-access-qtc58\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:22:37.416563 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.416561    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/722a5aa0-74ba-479e-91d4-e5db4b6b91e4-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:22:37.416830 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.416800    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84084650-0c60-412c-af89-602e1b8a8101-error-404-isvc-fd7d8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-fd7d8-kube-rbac-proxy-sar-config") pod "84084650-0c60-412c-af89-602e1b8a8101" (UID: "84084650-0c60-412c-af89-602e1b8a8101"). InnerVolumeSpecName "error-404-isvc-fd7d8-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:22:37.418468 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.418441    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84084650-0c60-412c-af89-602e1b8a8101-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "84084650-0c60-412c-af89-602e1b8a8101" (UID: "84084650-0c60-412c-af89-602e1b8a8101"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:22:37.418468 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.418458    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84084650-0c60-412c-af89-602e1b8a8101-kube-api-access-zvwxk" (OuterVolumeSpecName: "kube-api-access-zvwxk") pod "84084650-0c60-412c-af89-602e1b8a8101" (UID: "84084650-0c60-412c-af89-602e1b8a8101"). InnerVolumeSpecName "kube-api-access-zvwxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:22:37.445991 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.445969    2568 generic.go:358] "Generic (PLEG): container finished" podID="722a5aa0-74ba-479e-91d4-e5db4b6b91e4" containerID="4d3d68a20fc9274367ea66fb3f9a8a2148f849f030fd63743e569e10f1197ed2" exitCode=0
May 20 16:22:37.446110 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.446071    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"
May 20 16:22:37.446110 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.446078    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56" event={"ID":"722a5aa0-74ba-479e-91d4-e5db4b6b91e4","Type":"ContainerDied","Data":"4d3d68a20fc9274367ea66fb3f9a8a2148f849f030fd63743e569e10f1197ed2"}
May 20 16:22:37.446224 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.446119    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56" event={"ID":"722a5aa0-74ba-479e-91d4-e5db4b6b91e4","Type":"ContainerDied","Data":"645550bfaf1ea41100a9af8c1a644ffaafe504a62abc6b950607e85782d2752c"}
May 20 16:22:37.446224 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.446139    2568 scope.go:117] "RemoveContainer" containerID="ee01c263670252341de6fa275890ae9fd5b017badf97090fe6879b0c77d7fa06"
May 20 16:22:37.447626 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.447602    2568 generic.go:358] "Generic (PLEG): container finished" podID="84084650-0c60-412c-af89-602e1b8a8101" containerID="859cd27f334a62a8392894775dd886b43e3b9ee7efb9513358d9adc9fd683890" exitCode=0
May 20 16:22:37.447736 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.447630    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24" event={"ID":"84084650-0c60-412c-af89-602e1b8a8101","Type":"ContainerDied","Data":"859cd27f334a62a8392894775dd886b43e3b9ee7efb9513358d9adc9fd683890"}
May 20 16:22:37.447736 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.447665    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24" event={"ID":"84084650-0c60-412c-af89-602e1b8a8101","Type":"ContainerDied","Data":"cab6b50dc29305cc42b1ddaa18708f19baa1bd1f12719b03b5f29ee6f8d89370"}
May 20 16:22:37.447736 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.447692    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"
May 20 16:22:37.455779 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.455761    2568 scope.go:117] "RemoveContainer" containerID="4d3d68a20fc9274367ea66fb3f9a8a2148f849f030fd63743e569e10f1197ed2"
May 20 16:22:37.463443 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.463420    2568 scope.go:117] "RemoveContainer" containerID="ee01c263670252341de6fa275890ae9fd5b017badf97090fe6879b0c77d7fa06"
May 20 16:22:37.463643 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:22:37.463625    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee01c263670252341de6fa275890ae9fd5b017badf97090fe6879b0c77d7fa06\": container with ID starting with ee01c263670252341de6fa275890ae9fd5b017badf97090fe6879b0c77d7fa06 not found: ID does not exist" containerID="ee01c263670252341de6fa275890ae9fd5b017badf97090fe6879b0c77d7fa06"
May 20 16:22:37.463708 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.463648    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee01c263670252341de6fa275890ae9fd5b017badf97090fe6879b0c77d7fa06"} err="failed to get container status \"ee01c263670252341de6fa275890ae9fd5b017badf97090fe6879b0c77d7fa06\": rpc error: code = NotFound desc = could not find container \"ee01c263670252341de6fa275890ae9fd5b017badf97090fe6879b0c77d7fa06\": container with ID starting with ee01c263670252341de6fa275890ae9fd5b017badf97090fe6879b0c77d7fa06 not found: ID does not exist"
May 20 16:22:37.463708 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.463664    2568 scope.go:117] "RemoveContainer" containerID="4d3d68a20fc9274367ea66fb3f9a8a2148f849f030fd63743e569e10f1197ed2"
May 20 16:22:37.463888 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:22:37.463867    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d3d68a20fc9274367ea66fb3f9a8a2148f849f030fd63743e569e10f1197ed2\": container with ID starting with 4d3d68a20fc9274367ea66fb3f9a8a2148f849f030fd63743e569e10f1197ed2 not found: ID does not exist" containerID="4d3d68a20fc9274367ea66fb3f9a8a2148f849f030fd63743e569e10f1197ed2"
May 20 16:22:37.463956 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.463893    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d3d68a20fc9274367ea66fb3f9a8a2148f849f030fd63743e569e10f1197ed2"} err="failed to get container status \"4d3d68a20fc9274367ea66fb3f9a8a2148f849f030fd63743e569e10f1197ed2\": rpc error: code = NotFound desc = could not find container \"4d3d68a20fc9274367ea66fb3f9a8a2148f849f030fd63743e569e10f1197ed2\": container with ID starting with 4d3d68a20fc9274367ea66fb3f9a8a2148f849f030fd63743e569e10f1197ed2 not found: ID does not exist"
May 20 16:22:37.463956 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.463909    2568 scope.go:117] "RemoveContainer" containerID="9679969d6747a4ab2bce30dcd8ed1a01a3bc7167f684a9ece4539c5e462a72f8"
May 20 16:22:37.471702 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.471597    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"]
May 20 16:22:37.471767 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.471714    2568 scope.go:117] "RemoveContainer" containerID="859cd27f334a62a8392894775dd886b43e3b9ee7efb9513358d9adc9fd683890"
May 20 16:22:37.471888 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:22:37.471870    2568 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722a5aa0_74ba_479e_91d4_e5db4b6b91e4.slice/crio-645550bfaf1ea41100a9af8c1a644ffaafe504a62abc6b950607e85782d2752c\": RecentStats: unable to find data in memory cache]"
May 20 16:22:37.475698 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.475677    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fd7d8-predictor-76dd8cbf6-7rp56"]
May 20 16:22:37.479103 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.479082    2568 scope.go:117] "RemoveContainer" containerID="9679969d6747a4ab2bce30dcd8ed1a01a3bc7167f684a9ece4539c5e462a72f8"
May 20 16:22:37.479338 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:22:37.479322    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9679969d6747a4ab2bce30dcd8ed1a01a3bc7167f684a9ece4539c5e462a72f8\": container with ID starting with 9679969d6747a4ab2bce30dcd8ed1a01a3bc7167f684a9ece4539c5e462a72f8 not found: ID does not exist" containerID="9679969d6747a4ab2bce30dcd8ed1a01a3bc7167f684a9ece4539c5e462a72f8"
May 20 16:22:37.479402 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.479342    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9679969d6747a4ab2bce30dcd8ed1a01a3bc7167f684a9ece4539c5e462a72f8"} err="failed to get container status \"9679969d6747a4ab2bce30dcd8ed1a01a3bc7167f684a9ece4539c5e462a72f8\": rpc error: code = NotFound desc = could not find container \"9679969d6747a4ab2bce30dcd8ed1a01a3bc7167f684a9ece4539c5e462a72f8\": container with ID starting with 9679969d6747a4ab2bce30dcd8ed1a01a3bc7167f684a9ece4539c5e462a72f8 not found: ID does not exist"
May 20 16:22:37.479402 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.479358    2568 scope.go:117] "RemoveContainer" containerID="859cd27f334a62a8392894775dd886b43e3b9ee7efb9513358d9adc9fd683890"
May 20 16:22:37.479584 ip-10-0-129-117 kubenswrapper[2568]: E0520 16:22:37.479567    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"859cd27f334a62a8392894775dd886b43e3b9ee7efb9513358d9adc9fd683890\": container with ID starting with 859cd27f334a62a8392894775dd886b43e3b9ee7efb9513358d9adc9fd683890 not found: ID does not exist" containerID="859cd27f334a62a8392894775dd886b43e3b9ee7efb9513358d9adc9fd683890"
May 20 16:22:37.479625 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.479590    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"859cd27f334a62a8392894775dd886b43e3b9ee7efb9513358d9adc9fd683890"} err="failed to get container status \"859cd27f334a62a8392894775dd886b43e3b9ee7efb9513358d9adc9fd683890\": rpc error: code = NotFound desc = could not find container \"859cd27f334a62a8392894775dd886b43e3b9ee7efb9513358d9adc9fd683890\": container with ID starting with 859cd27f334a62a8392894775dd886b43e3b9ee7efb9513358d9adc9fd683890 not found: ID does not exist"
May 20 16:22:37.492517 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.492463    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"]
May 20 16:22:37.498029 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.498000    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fd7d8-predictor-6dfb6b7f9-72m24"]
May 20 16:22:37.517794 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.517773    2568 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-fd7d8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84084650-0c60-412c-af89-602e1b8a8101-error-404-isvc-fd7d8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:22:37.517864 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.517795    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvwxk\" (UniqueName: \"kubernetes.io/projected/84084650-0c60-412c-af89-602e1b8a8101-kube-api-access-zvwxk\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:22:37.517864 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.517806    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84084650-0c60-412c-af89-602e1b8a8101-proxy-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 16:22:37.643118 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.643092    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="722a5aa0-74ba-479e-91d4-e5db4b6b91e4" path="/var/lib/kubelet/pods/722a5aa0-74ba-479e-91d4-e5db4b6b91e4/volumes"
May 20 16:22:37.643487 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:22:37.643475    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84084650-0c60-412c-af89-602e1b8a8101" path="/var/lib/kubelet/pods/84084650-0c60-412c-af89-602e1b8a8101/volumes"
May 20 16:23:06.708954 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.708922    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kp2hs/must-gather-xsmbn"]
May 20 16:23:06.709570 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709406    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4ac65ba-473b-4af6-bfc2-bce8092a55ce" containerName="kube-rbac-proxy"
May 20 16:23:06.709570 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709425    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4ac65ba-473b-4af6-bfc2-bce8092a55ce" containerName="kube-rbac-proxy"
May 20 16:23:06.709570 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709443    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84084650-0c60-412c-af89-602e1b8a8101" containerName="kube-rbac-proxy"
May 20 16:23:06.709570 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709452    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="84084650-0c60-412c-af89-602e1b8a8101" containerName="kube-rbac-proxy"
May 20 16:23:06.709570 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709465    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="722a5aa0-74ba-479e-91d4-e5db4b6b91e4" containerName="kube-rbac-proxy"
May 20 16:23:06.709570 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709474    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="722a5aa0-74ba-479e-91d4-e5db4b6b91e4" containerName="kube-rbac-proxy"
May 20 16:23:06.709570 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709486    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eda94f7-0a45-42af-ace9-a99be2b1368c" containerName="kube-rbac-proxy"
May 20 16:23:06.709570 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709494    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eda94f7-0a45-42af-ace9-a99be2b1368c" containerName="kube-rbac-proxy"
May 20 16:23:06.709570 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709507    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="722a5aa0-74ba-479e-91d4-e5db4b6b91e4" containerName="kserve-container"
May 20 16:23:06.709570 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709515    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="722a5aa0-74ba-479e-91d4-e5db4b6b91e4" containerName="kserve-container"
May 20 16:23:06.709570 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709527    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84084650-0c60-412c-af89-602e1b8a8101" containerName="kserve-container"
May 20 16:23:06.709570 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709534    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="84084650-0c60-412c-af89-602e1b8a8101" containerName="kserve-container"
May 20 16:23:06.709570 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709551    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4ac65ba-473b-4af6-bfc2-bce8092a55ce" containerName="kserve-container"
May 20 16:23:06.709570 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709559    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4ac65ba-473b-4af6-bfc2-bce8092a55ce" containerName="kserve-container"
May 20 16:23:06.709570 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709570    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92995157-1adc-4560-ae14-06cd7783fa6c" containerName="kserve-container"
May 20 16:23:06.709570 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709578    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="92995157-1adc-4560-ae14-06cd7783fa6c" containerName="kserve-container"
May 20 16:23:06.710383 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709589    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eda94f7-0a45-42af-ace9-a99be2b1368c" containerName="kserve-container"
May 20 16:23:06.710383 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709596    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eda94f7-0a45-42af-ace9-a99be2b1368c" containerName="kserve-container"
May 20 16:23:06.710383 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709605    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92995157-1adc-4560-ae14-06cd7783fa6c" containerName="kube-rbac-proxy"
May 20 16:23:06.710383 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709612    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="92995157-1adc-4560-ae14-06cd7783fa6c" containerName="kube-rbac-proxy"
May 20 16:23:06.710383 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709628    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed" containerName="kube-rbac-proxy"
May 20 16:23:06.710383 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709636    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed" containerName="kube-rbac-proxy"
May 20 16:23:06.710383 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709647    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed" containerName="kserve-container"
May 20 16:23:06.710383 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709655    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed" containerName="kserve-container"
May 20 16:23:06.710383 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709732    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="0eda94f7-0a45-42af-ace9-a99be2b1368c" containerName="kserve-container"
May 20 16:23:06.710383 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709744    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="f4ac65ba-473b-4af6-bfc2-bce8092a55ce" containerName="kube-rbac-proxy"
May 20 16:23:06.710383 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709754    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="84084650-0c60-412c-af89-602e1b8a8101" containerName="kube-rbac-proxy"
May 20 16:23:06.710383 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709768    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed" containerName="kserve-container"
May 20 16:23:06.710383 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709777    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="92995157-1adc-4560-ae14-06cd7783fa6c" containerName="kube-rbac-proxy"
May 20 16:23:06.710383 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709787    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="ffc0ed77-16ae-45e9-b1e7-19f1c4b44fed" containerName="kube-rbac-proxy"
May 20 16:23:06.710383 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709797    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="722a5aa0-74ba-479e-91d4-e5db4b6b91e4" containerName="kube-rbac-proxy"
May 20 16:23:06.710383 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709806    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="722a5aa0-74ba-479e-91d4-e5db4b6b91e4" containerName="kserve-container"
May 20 16:23:06.710383 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709815    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="92995157-1adc-4560-ae14-06cd7783fa6c" containerName="kserve-container"
May 20 16:23:06.710383 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709826    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="0eda94f7-0a45-42af-ace9-a99be2b1368c" containerName="kube-rbac-proxy"
May 20 16:23:06.710383 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709836    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="f4ac65ba-473b-4af6-bfc2-bce8092a55ce" containerName="kserve-container"
May 20 16:23:06.710383 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.709845    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="84084650-0c60-412c-af89-602e1b8a8101" containerName="kserve-container"
May 20 16:23:06.713159 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.713141    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kp2hs/must-gather-xsmbn"
May 20 16:23:06.715472 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.715449    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kp2hs\"/\"kube-root-ca.crt\""
May 20 16:23:06.715589 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.715472    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kp2hs\"/\"openshift-service-ca.crt\""
May 20 16:23:06.715589 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.715528    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kp2hs\"/\"default-dockercfg-qjmg8\""
May 20 16:23:06.721432 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.721403    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kp2hs/must-gather-xsmbn"]
May 20 16:23:06.841137 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.841098    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/99db988a-4008-408b-831a-da8ba28b3ae8-must-gather-output\") pod \"must-gather-xsmbn\" (UID: \"99db988a-4008-408b-831a-da8ba28b3ae8\") " pod="openshift-must-gather-kp2hs/must-gather-xsmbn"
May 20 16:23:06.841294 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.841156    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgv4h\" (UniqueName: \"kubernetes.io/projected/99db988a-4008-408b-831a-da8ba28b3ae8-kube-api-access-jgv4h\") pod \"must-gather-xsmbn\" (UID: \"99db988a-4008-408b-831a-da8ba28b3ae8\") " pod="openshift-must-gather-kp2hs/must-gather-xsmbn"
May 20 16:23:06.942524 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.942478    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/99db988a-4008-408b-831a-da8ba28b3ae8-must-gather-output\") pod \"must-gather-xsmbn\" (UID: \"99db988a-4008-408b-831a-da8ba28b3ae8\") " pod="openshift-must-gather-kp2hs/must-gather-xsmbn"
May 20 16:23:06.942697 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.942540    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jgv4h\" (UniqueName: \"kubernetes.io/projected/99db988a-4008-408b-831a-da8ba28b3ae8-kube-api-access-jgv4h\") pod \"must-gather-xsmbn\" (UID: \"99db988a-4008-408b-831a-da8ba28b3ae8\") " pod="openshift-must-gather-kp2hs/must-gather-xsmbn"
May 20 16:23:06.942819 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.942798    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/99db988a-4008-408b-831a-da8ba28b3ae8-must-gather-output\") pod \"must-gather-xsmbn\" (UID: \"99db988a-4008-408b-831a-da8ba28b3ae8\") " pod="openshift-must-gather-kp2hs/must-gather-xsmbn"
May 20 16:23:06.951818 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:06.951789    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgv4h\" (UniqueName: \"kubernetes.io/projected/99db988a-4008-408b-831a-da8ba28b3ae8-kube-api-access-jgv4h\") pod \"must-gather-xsmbn\" (UID: \"99db988a-4008-408b-831a-da8ba28b3ae8\") " pod="openshift-must-gather-kp2hs/must-gather-xsmbn"
May 20 16:23:07.032183 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:07.032115    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kp2hs/must-gather-xsmbn"
May 20 16:23:07.162972 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:07.162950    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kp2hs/must-gather-xsmbn"]
May 20 16:23:07.165010 ip-10-0-129-117 kubenswrapper[2568]: W0520 16:23:07.164970    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99db988a_4008_408b_831a_da8ba28b3ae8.slice/crio-582ad6ce2370227fdebfdca41d686cd1431541ae6b0b066ab327187b7581d615 WatchSource:0}: Error finding container 582ad6ce2370227fdebfdca41d686cd1431541ae6b0b066ab327187b7581d615: Status 404 returned error can't find the container with id 582ad6ce2370227fdebfdca41d686cd1431541ae6b0b066ab327187b7581d615
May 20 16:23:07.166858 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:07.166840    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 16:23:07.548783 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:07.548755    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kp2hs/must-gather-xsmbn" event={"ID":"99db988a-4008-408b-831a-da8ba28b3ae8","Type":"ContainerStarted","Data":"582ad6ce2370227fdebfdca41d686cd1431541ae6b0b066ab327187b7581d615"}
May 20 16:23:08.554596 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:08.554554    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kp2hs/must-gather-xsmbn" event={"ID":"99db988a-4008-408b-831a-da8ba28b3ae8","Type":"ContainerStarted","Data":"3661f8f44f762f534b10ab2e13351ee58e730d340b751ac1ca5c69569b7f529e"}
May 20 16:23:08.554982 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:08.554601    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kp2hs/must-gather-xsmbn" event={"ID":"99db988a-4008-408b-831a-da8ba28b3ae8","Type":"ContainerStarted","Data":"bf9b44a172b81cb402e6e41d04c5f97a4da191d45475499311639a7ce4141de5"}
May 20 16:23:08.571001 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:08.570953    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-kp2hs/must-gather-xsmbn" podStartSLOduration=1.625973731 podStartE2EDuration="2.57093877s" podCreationTimestamp="2026-05-20 16:23:06 +0000 UTC" firstStartedPulling="2026-05-20 16:23:07.166995844 +0000 UTC m=+3096.105837379" lastFinishedPulling="2026-05-20 16:23:08.111960868 +0000 UTC m=+3097.050802418" observedRunningTime="2026-05-20 16:23:08.56947942 +0000 UTC m=+3097.508320979" watchObservedRunningTime="2026-05-20 16:23:08.57093877 +0000 UTC m=+3097.509780327"
May 20 16:23:09.526430 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:09.526396    2568 ???:1] "http: TLS handshake error from 10.0.129.117:49838: EOF"
May 20 16:23:09.533810 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:09.533782    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cknlk_cb6d0d3d-79d3-4f3f-8f8e-974108118563/global-pull-secret-syncer/0.log"
May 20 16:23:09.669161 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:09.669133    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g8s5k_f5adfb93-88e3-4b63-839c-31c7a0793b04/konnectivity-agent/0.log"
May 20 16:23:09.757910 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:09.757875    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-117.ec2.internal_6f81a807c436a78d0ae12cc5e66b2e05/haproxy/0.log"
May 20 16:23:13.250498 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:13.250459    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6c75959455-c85d6_e984d532-c6fd-4d79-8b16-07fc216f278d/cluster-monitoring-operator/0.log"
May 20 16:23:13.279676 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:13.279648    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-nfcxl_9ddf79fd-adcf-46d4-b6cf-27f541a98e08/kube-state-metrics/0.log"
May 20 16:23:13.305205 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:13.305173    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-nfcxl_9ddf79fd-adcf-46d4-b6cf-27f541a98e08/kube-rbac-proxy-main/0.log"
May 20 16:23:13.329668 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:13.329638    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-nfcxl_9ddf79fd-adcf-46d4-b6cf-27f541a98e08/kube-rbac-proxy-self/0.log"
May 20 16:23:13.513596 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:13.513512    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gd2xt_6be29376-46fb-484c-87de-2a74cb66ce76/node-exporter/0.log"
May 20 16:23:13.536506 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:13.536474    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gd2xt_6be29376-46fb-484c-87de-2a74cb66ce76/kube-rbac-proxy/0.log"
May 20 16:23:13.560916 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:13.560886    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gd2xt_6be29376-46fb-484c-87de-2a74cb66ce76/init-textfile/0.log"
May 20 16:23:13.676155 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:13.676126    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-8k4jn_52f2ef1f-4690-4c87-8acd-ded5750e21e4/kube-rbac-proxy-main/0.log"
May 20 16:23:13.700193 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:13.700152    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-8k4jn_52f2ef1f-4690-4c87-8acd-ded5750e21e4/kube-rbac-proxy-self/0.log"
May 20 16:23:13.725040 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:13.724983    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-8k4jn_52f2ef1f-4690-4c87-8acd-ded5750e21e4/openshift-state-metrics/0.log"
May 20 16:23:13.938311 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:13.938276    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-4g7z6_ef8af763-035c-4a9a-9f38-91d2dcc95823/prometheus-operator/0.log"
May 20 16:23:13.962359 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:13.962319    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-4g7z6_ef8af763-035c-4a9a-9f38-91d2dcc95823/kube-rbac-proxy/0.log"
May 20 16:23:15.409621 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:15.409590    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-d4d8f94f9-2fdvj_94373970-3fe5-4bbf-a05a-f4913b27e8d0/networking-console-plugin/0.log"
May 20 16:23:16.215650 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:16.215622    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-b95c64d84-z69xn_ecacce7e-932c-4c2a-9108-78652b56bcd8/console/0.log"
May 20 16:23:16.406376 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:16.406344    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2"]
May 20 16:23:16.411101 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:16.411082    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2"
May 20 16:23:16.418603 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:16.418579    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2"]
May 20 16:23:16.529968 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:16.529894    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b8c9339-e1e1-44a6-a934-97bf8ecfbf46-sys\") pod \"perf-node-gather-daemonset-7mjc2\" (UID: \"3b8c9339-e1e1-44a6-a934-97bf8ecfbf46\") " pod="openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2"
May 20 16:23:16.529968 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:16.529949    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b8c9339-e1e1-44a6-a934-97bf8ecfbf46-lib-modules\") pod \"perf-node-gather-daemonset-7mjc2\" (UID: \"3b8c9339-e1e1-44a6-a934-97bf8ecfbf46\") " pod="openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2"
May 20 16:23:16.530173 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:16.529992    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3b8c9339-e1e1-44a6-a934-97bf8ecfbf46-proc\") pod \"perf-node-gather-daemonset-7mjc2\" (UID: \"3b8c9339-e1e1-44a6-a934-97bf8ecfbf46\") " pod="openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2"
May 20 16:23:16.530173 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:16.530045    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztp6w\" (UniqueName: \"kubernetes.io/projected/3b8c9339-e1e1-44a6-a934-97bf8ecfbf46-kube-api-access-ztp6w\") pod \"perf-node-gather-daemonset-7mjc2\" (UID: \"3b8c9339-e1e1-44a6-a934-97bf8ecfbf46\") " pod="openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2"
May 20 16:23:16.530173 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:16.530068    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3b8c9339-e1e1-44a6-a934-97bf8ecfbf46-podres\") pod \"perf-node-gather-daemonset-7mjc2\" (UID: \"3b8c9339-e1e1-44a6-a934-97bf8ecfbf46\") " pod="openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2"
May 20 16:23:16.632297 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:16.631343    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b8c9339-e1e1-44a6-a934-97bf8ecfbf46-lib-modules\") pod \"perf-node-gather-daemonset-7mjc2\" (UID: \"3b8c9339-e1e1-44a6-a934-97bf8ecfbf46\") " pod="openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2"
May 20 16:23:16.632297 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:16.631425    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3b8c9339-e1e1-44a6-a934-97bf8ecfbf46-proc\") pod \"perf-node-gather-daemonset-7mjc2\" (UID: \"3b8c9339-e1e1-44a6-a934-97bf8ecfbf46\") " pod="openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2"
May 20 16:23:16.632297 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:16.631476    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ztp6w\" (UniqueName: \"kubernetes.io/projected/3b8c9339-e1e1-44a6-a934-97bf8ecfbf46-kube-api-access-ztp6w\") pod \"perf-node-gather-daemonset-7mjc2\" (UID: \"3b8c9339-e1e1-44a6-a934-97bf8ecfbf46\") " pod="openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2"
May 20 16:23:16.632297 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:16.631507    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3b8c9339-e1e1-44a6-a934-97bf8ecfbf46-podres\") pod \"perf-node-gather-daemonset-7mjc2\" (UID: \"3b8c9339-e1e1-44a6-a934-97bf8ecfbf46\") " pod="openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2"
May 20 16:23:16.632297 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:16.631575    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b8c9339-e1e1-44a6-a934-97bf8ecfbf46-sys\") pod \"perf-node-gather-daemonset-7mjc2\" (UID: \"3b8c9339-e1e1-44a6-a934-97bf8ecfbf46\") " pod="openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2"
May 20 16:23:16.632297 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:16.631668    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b8c9339-e1e1-44a6-a934-97bf8ecfbf46-sys\") pod \"perf-node-gather-daemonset-7mjc2\" (UID: \"3b8c9339-e1e1-44a6-a934-97bf8ecfbf46\") " pod="openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2"
May 20 16:23:16.632297 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:16.631791    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b8c9339-e1e1-44a6-a934-97bf8ecfbf46-lib-modules\") pod \"perf-node-gather-daemonset-7mjc2\" (UID: \"3b8c9339-e1e1-44a6-a934-97bf8ecfbf46\") " pod="openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2"
May 20 16:23:16.632297 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:16.631844    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3b8c9339-e1e1-44a6-a934-97bf8ecfbf46-proc\") pod \"perf-node-gather-daemonset-7mjc2\" (UID: \"3b8c9339-e1e1-44a6-a934-97bf8ecfbf46\") " pod="openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2"
May 20 16:23:16.632297 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:16.632255    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3b8c9339-e1e1-44a6-a934-97bf8ecfbf46-podres\") pod \"perf-node-gather-daemonset-7mjc2\" (UID: \"3b8c9339-e1e1-44a6-a934-97bf8ecfbf46\") " pod="openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2"
May 20 16:23:16.640130 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:16.640105    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztp6w\" (UniqueName: \"kubernetes.io/projected/3b8c9339-e1e1-44a6-a934-97bf8ecfbf46-kube-api-access-ztp6w\") pod \"perf-node-gather-daemonset-7mjc2\" (UID: \"3b8c9339-e1e1-44a6-a934-97bf8ecfbf46\") " pod="openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2"
May 20 16:23:16.666258 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:16.666238    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5cf8c89d64-lz659_3390fc6f-b1ea-40b1-b920-87f3f2f96a12/volume-data-source-validator/0.log"
May 20 16:23:16.723481 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:16.723455    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2"
May 20 16:23:16.870249 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:16.870218    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2"]
May 20 16:23:16.873997 ip-10-0-129-117 kubenswrapper[2568]: W0520 16:23:16.873777    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3b8c9339_e1e1_44a6_a934_97bf8ecfbf46.slice/crio-3941d3673341b943b65ee3505550666ef9d922324013c7cdddc8c79ce1b432d1 WatchSource:0}: Error finding container 3941d3673341b943b65ee3505550666ef9d922324013c7cdddc8c79ce1b432d1: Status 404 returned error can't find the container with id 3941d3673341b943b65ee3505550666ef9d922324013c7cdddc8c79ce1b432d1
May 20 16:23:17.435058 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:17.434994    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w8bsv_9bcdce67-2f01-4c5e-a20f-c976159f8683/dns/0.log"
May 20 16:23:17.464889 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:17.464871    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w8bsv_9bcdce67-2f01-4c5e-a20f-c976159f8683/kube-rbac-proxy/0.log"
May 20 16:23:17.577390 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:17.577345    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vkh2n_abd4b634-19df-4a65-b43b-f3cd86482afe/dns-node-resolver/0.log"
May 20 16:23:17.596536 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:17.596499    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2" event={"ID":"3b8c9339-e1e1-44a6-a934-97bf8ecfbf46","Type":"ContainerStarted","Data":"4e708225b6ed44b40769447f7712ec1b58ba00439b399290e15a33df1fe83128"}
May 20 16:23:17.596536 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:17.596550    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2" event={"ID":"3b8c9339-e1e1-44a6-a934-97bf8ecfbf46","Type":"ContainerStarted","Data":"3941d3673341b943b65ee3505550666ef9d922324013c7cdddc8c79ce1b432d1"}
May 20 16:23:17.596785 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:17.596588    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2"
May 20 16:23:18.082033 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:18.081982    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l24nh_2fe4adfc-e1dd-4a70-a161-f0d525f3ca3b/node-ca/0.log"
May 20 16:23:19.255830 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:19.255805    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-szz5l_fbac1f01-44d4-4042-a375-5a6a14f94d47/serve-healthcheck-canary/0.log"
May 20 16:23:19.637389 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:19.637362    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-8ltwr_0cbb5691-4ce1-46dd-8a53-d94b5c19edcf/insights-operator/0.log"
May 20 16:23:19.638679 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:19.638657    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-8ltwr_0cbb5691-4ce1-46dd-8a53-d94b5c19edcf/insights-operator/1.log"
May 20 16:23:19.806376 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:19.806348    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w8wdt_f44dd61a-10fc-4e25-ba59-ae239dd254e5/kube-rbac-proxy/0.log"
May 20 16:23:19.841160 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:19.841137    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w8wdt_f44dd61a-10fc-4e25-ba59-ae239dd254e5/exporter/0.log"
May 20 16:23:19.881038 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:19.880986    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w8wdt_f44dd61a-10fc-4e25-ba59-ae239dd254e5/extractor/0.log"
May 20 16:23:21.821447 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:21.821404    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-87b4b9896-c4cbk_c9af9950-cb53-4e78-a702-7cc8b21f4472/manager/0.log"
May 20 16:23:22.125618 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:22.125551    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-pzwkl_e006510d-31f4-435c-9001-eaae8ed6b5e5/s3-init/0.log"
May 20 16:23:22.155294 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:22.155271    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-79f457656b-4pzks_ddecec2b-9f2a-4742-963b-6d272fcb4308/seaweedfs/0.log"
May 20 16:23:23.611434 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:23.611393    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2"
May 20 16:23:23.628420 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:23.628363    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-kp2hs/perf-node-gather-daemonset-7mjc2" podStartSLOduration=7.628345274 podStartE2EDuration="7.628345274s" podCreationTimestamp="2026-05-20 16:23:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:23:17.613116627 +0000 UTC m=+3106.551958185" watchObservedRunningTime="2026-05-20 16:23:23.628345274 +0000 UTC m=+3112.567186834"
May 20 16:23:26.139940 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:26.139887    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-dbg6m_1ba2f334-7a2e-4fff-a6e7-7590c25e1dae/migrator/0.log"
May 20 16:23:26.165803 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:26.165780    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-dbg6m_1ba2f334-7a2e-4fff-a6e7-7590c25e1dae/graceful-termination/0.log"
May 20 16:23:26.512068 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:26.511945    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-mbpgr_1bd01bf5-75e6-45d5-9bf3-0b480b5457e8/kube-storage-version-migrator-operator/1.log"
May 20 16:23:26.512809 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:26.512788    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-mbpgr_1bd01bf5-75e6-45d5-9bf3-0b480b5457e8/kube-storage-version-migrator-operator/0.log"
May 20 16:23:27.653562 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:27.653524    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kvrb6_799a84c1-8372-4b99-a34d-105048106d5d/kube-multus-additional-cni-plugins/0.log"
May 20 16:23:27.678419 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:27.678392    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kvrb6_799a84c1-8372-4b99-a34d-105048106d5d/egress-router-binary-copy/0.log"
May 20 16:23:27.703702 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:27.703648    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kvrb6_799a84c1-8372-4b99-a34d-105048106d5d/cni-plugins/0.log"
May 20 16:23:27.730422 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:27.730400    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kvrb6_799a84c1-8372-4b99-a34d-105048106d5d/bond-cni-plugin/0.log"
May 20 16:23:27.752010 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:27.751985    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kvrb6_799a84c1-8372-4b99-a34d-105048106d5d/routeoverride-cni/0.log"
May 20 16:23:27.774684 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:27.774655    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kvrb6_799a84c1-8372-4b99-a34d-105048106d5d/whereabouts-cni-bincopy/0.log"
May 20 16:23:27.797633 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:27.797606    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kvrb6_799a84c1-8372-4b99-a34d-105048106d5d/whereabouts-cni/0.log"
May 20 16:23:28.002051 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:28.001955    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bjx28_d3f5b3cf-5e4f-49d9-ae81-f81badb12283/kube-multus/0.log"
May 20 16:23:28.147038 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:28.146997    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-skk8q_3022a27a-6112-46ee-a54b-c5af5bee2a1f/network-metrics-daemon/0.log"
May 20 16:23:28.166526 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:28.166496    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-skk8q_3022a27a-6112-46ee-a54b-c5af5bee2a1f/kube-rbac-proxy/0.log"
May 20 16:23:28.892731 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:28.892690    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-controller/0.log"
May 20 16:23:28.914058 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:28.913998    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/0.log"
May 20 16:23:28.928961 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:28.928922    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovn-acl-logging/1.log"
May 20 16:23:28.954725 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:28.954695    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/kube-rbac-proxy-node/0.log"
May 20 16:23:28.978361 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:28.978332    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/kube-rbac-proxy-ovn-metrics/0.log"
May 20 16:23:28.998883 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:28.998854    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/northd/0.log"
May 20 16:23:29.021843 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:29.021815    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/nbdb/0.log"
May 20 16:23:29.045877 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:29.045846    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/sbdb/0.log"
May 20 16:23:29.163962 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:29.163874    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gqhp4_6bf94e80-c129-435d-a864-29e97ca3634f/ovnkube-controller/0.log"
May 20 16:23:30.808226 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:30.808194    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7f44b6dd87-7crcg_74f685e9-2b4e-47a2-b2e0-dd6158968cb3/check-endpoints/0.log"
May 20 16:23:30.884818 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:30.884794    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-r9x8t_69c7e426-cc15-4efc-afbe-77377c1c3779/network-check-target-container/0.log"
May 20 16:23:31.827098 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:31.827063    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-kffdt_6fefe56a-44d8-47d6-bae5-3926f0bc3b1c/iptables-alerter/0.log"
May 20 16:23:32.543719 ip-10-0-129-117 kubenswrapper[2568]: I0520 16:23:32.543691    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-th4j4_3ccfe95d-efc9-46ec-aaac-da92e41a5574/tuned/0.log"